2024-04-25T03:05:33.057 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T03:05:33.058 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443 --description krbd/mirror/{bluestore-bitmap clusters/2-node conf install/ceph ms_mode$/{crc-rxbounce} tasks/compare-mirror-images} --verbose --name teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/orig.config.yaml 2024-04-25T03:05:33.093 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T03:05:33.175 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443 branch: squid description: krbd/mirror/{bluestore-bitmap clusters/2-node conf install/ceph ms_mode$/{crc-rxbounce} tasks/compare-mirror-images} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672443' kernel: flavor: default kdb: 1 sha1: 806ec36b21129f113ce356c007dd59694d340b37 last_in_suite: false machine_type: smithi meta: - desc: 2 ceph clusters with 1 mon, 1 mgr and 3 osd each name: teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: squid ceph: conf: client: rbd default map options: ms_mode=crc,rxbounce global: mon warn on pool no app: false ms die on skipped message: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: extra_system_packages: - pv flavor: default sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - - cluster2.mon.a - cluster2.mgr.x - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster2.client.mirror - cluster2.client.mirror.0 seed: 3056 sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 sleep_before_teardown: 0 subset: 0/4 suite: krbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 targets: smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEE4UpcTt0SOnbcfyMKwV4DsXIWvHxfC6eU1pVdzrNrg6NXt4UllBl9EMkpqFCKfJ7+7WOztKYtA8MjaHOAAGC0= smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKLEC0jIcC1bw/OnyBAENWc1L9joCrlM1lzqES0/EyFKQf9xlK/0cgUcDhqfNW8nF/7ttwllaF53FrMpKlNFiCg= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - rbd-mirror: client: cluster1.client.mirror.0 thrash: false - rbd-mirror: client: cluster2.client.mirror.0 thrash: false - workunit: clients: cluster1.client.mirror: - rbd/compare_mirror_images.sh env: RBD_DEVICE_TYPE: krbd RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 3h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:48:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T03:05:33.175 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-25T03:05:33.176 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-25T03:05:33.176 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T03:05:33.178 INFO:teuthology.task.internal:Checking packages... 2024-04-25T03:05:33.207 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b98aab6c00f483888c049d0d8e7bc507e115dd86' 2024-04-25T03:05:33.207 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T03:05:33.207 INFO:teuthology.packaging:ref: None 2024-04-25T03:05:33.207 INFO:teuthology.packaging:tag: None 2024-04-25T03:05:33.207 INFO:teuthology.packaging:branch: squid 2024-04-25T03:05:33.207 INFO:teuthology.packaging:sha1: b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-25T03:05:33.208 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=squid 2024-04-25T03:05:33.471 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-25T03:05:33.474 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T03:05:33.485 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T03:05:33.485 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T03:05:33.528 INFO:teuthology.task.internal:Saving configuration 2024-04-25T03:05:33.541 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T03:05:33.554 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T03:05:33.579 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 03:00:25.397892', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEE4UpcTt0SOnbcfyMKwV4DsXIWvHxfC6eU1pVdzrNrg6NXt4UllBl9EMkpqFCKfJ7+7WOztKYtA8MjaHOAAGC0='} 2024-04-25T03:05:33.602 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 03:00:25.399661', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKLEC0jIcC1bw/OnyBAENWc1L9joCrlM1lzqES0/EyFKQf9xlK/0cgUcDhqfNW8nF/7ttwllaF53FrMpKlNFiCg='} 2024-04-25T03:05:33.602 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T03:05:33.614 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2'] 2024-04-25T03:05:33.614 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.mgr.x', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster2.client.mirror', 'cluster2.client.mirror.0'] 2024-04-25T03:05:33.614 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T03:05:33.720 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3b2b166e50>, signals=[15]) 2024-04-25T03:05:33.720 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T03:05:33.732 INFO:teuthology.task.internal:Opening connections... 2024-04-25T03:05:33.733 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T03:05:33.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:05:33.815 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T03:05:33.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:05:33.890 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T03:05:33.901 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-04-25T03:05:33.924 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-04-25T03:05:33.924 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-04-25T03:05:33.986 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-04-25T03:05:33.986 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="8" 2024-04-25T03:05:33.986 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-04-25T03:05:33.986 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-04-25T03:05:33.986 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="8" 2024-04-25T03:05:33.986 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el8" 2024-04-25T03:05:33.987 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-25T03:05:33.987 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-04-25T03:05:33.987 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-25T03:05:33.987 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-04-25T03:05:33.987 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T03:05:33.987 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-25T03:05:33.987 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T03:05:33.988 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-04-25T03:05:34.009 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-04-25T03:05:34.031 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-04-25T03:05:34.031 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-25T03:05:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-04-25T03:05:34.094 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T03:05:34.094 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-25T03:05:34.094 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T03:05:34.094 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-04-25T03:05:34.117 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T03:05:34.128 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T03:05:34.195 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T03:05:34.195 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-04-25T03:05:34.198 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-04-25T03:05:34.217 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T03:05:34.231 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T03:05:34.231 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-04-25T03:05:34.259 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-04-25T03:05:34.290 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T03:05:34.451 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T03:05:34.468 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': 1, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-25T03:05:34.469 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'cluster2.mon.a': {'flavor': 'default', 'kdb': 1, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, timeout 300 2024-04-25T03:05:34.469 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T03:05:34.469 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T03:05:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T03:05:34.490 DEBUG:teuthology.task.kernel:sha1 for cluster2.mon.a is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:05:34.490 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-25T03:05:34.490 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-04-25T03:05:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T03:05:34.491 DEBUG:teuthology.task.kernel:sha1 for cluster1.mon.a is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:05:34.491 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-25T03:05:34.492 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-04-25T03:05:34.552 INFO:teuthology.orchestra.run.smithi188.stdout:4.18.0-489.el8.x86_64 2024-04-25T03:05:34.552 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:05:34.552 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-25T03:05:34.552 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:05:34.565 INFO:teuthology.orchestra.run.smithi173.stdout:4.18.0-489.el8.x86_64 2024-04-25T03:05:34.565 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-489.el8.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:05:34.565 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-25T03:05:34.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:05:34.777 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum upgrade -y linux-firmware 2024-04-25T03:05:34.783 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum upgrade -y linux-firmware 2024-04-25T03:05:37.880 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 42 MB/s | 28 MB 00:00 2024-04-25T03:05:38.234 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 32 MB/s | 28 MB 00:00 2024-04-25T03:05:43.463 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 17 MB/s | 10 MB 00:00 2024-04-25T03:05:43.900 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 15 MB/s | 10 MB 00:00 2024-04-25T03:05:45.463 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2024-04-25T03:05:45.717 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 85 kB/s | 18 kB 00:00 2024-04-25T03:05:45.756 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 43 kB/s | 7.7 kB 00:00 2024-04-25T03:05:46.016 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-04-25T03:05:46.168 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 7.7 kB 00:00 2024-04-25T03:05:46.386 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-04-25T03:05:47.016 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2024-04-25T03:05:48.270 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2024-04-25T03:05:50.833 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 421 kB/s | 24 kB 00:00 2024-04-25T03:05:52.012 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 447 kB/s | 24 kB 00:00 2024-04-25T03:05:54.025 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-25T03:05:54.025 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T03:05:54.025 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-04-25T03:05:54.025 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T03:05:54.025 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-04-25T03:05:54.025 INFO:teuthology.orchestra.run.smithi173.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-04-25T03:05:54.025 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:05:54.026 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-25T03:05:54.026 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T03:05:54.026 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 1 Package 2024-04-25T03:05:54.026 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:05:54.026 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 355 M 2024-04-25T03:05:54.026 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-04-25T03:05:55.254 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout: linux-firmware noarch 20240111-121.gitb3132c18.el8 baseos 355 M 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-25T03:05:55.255 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-04-25T03:05:55.256 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 1 Package 2024-04-25T03:05:55.256 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:05:55.256 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 355 M 2024-04-25T03:05:55.256 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-25T03:06:06.162 INFO:teuthology.orchestra.run.smithi173.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 29 MB/s | 355 MB 00:12 2024-04-25T03:06:06.162 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-04-25T03:06:06.162 INFO:teuthology.orchestra.run.smithi173.stdout:Total 29 MB/s | 355 MB 00:12 2024-04-25T03:06:08.260 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-04-25T03:06:08.314 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-04-25T03:06:08.314 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-04-25T03:06:10.492 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-04-25T03:06:10.509 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-04-25T03:06:19.959 INFO:teuthology.orchestra.run.smithi188.stdout:linux-firmware-20240111-121.gitb3132c18.el8.noa 14 MB/s | 355 MB 00:24 2024-04-25T03:06:19.959 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-25T03:06:19.959 INFO:teuthology.orchestra.run.smithi188.stdout:Total 14 MB/s | 355 MB 00:24 2024-04-25T03:06:22.064 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-25T03:06:22.145 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-25T03:06:22.145 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-25T03:06:24.307 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-25T03:06:24.331 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-25T03:06:33.531 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-04-25T03:06:33.884 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-25T03:06:34.121 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-25T03:06:34.360 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-25T03:06:34.360 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-25T03:06:34.829 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-25T03:06:34.829 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:06:34.829 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-04-25T03:06:34.829 INFO:teuthology.orchestra.run.smithi173.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-04-25T03:06:34.829 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:06:34.829 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-25T03:06:35.059 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on cluster1.mon.a... 2024-04-25T03:06:35.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:06:35.209 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/8/flavors/default/x86_64/ 2024-04-25T03:06:35.210 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/8/flavors/default/x86_64/ --input-file=- 2024-04-25T03:06:35.212 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi173.front.sepia.ceph.com to complete... 2024-04-25T03:06:47.072 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-25T03:06:47.405 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-25T03:06:47.674 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-25T03:06:47.917 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-25T03:06:47.917 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : linux-firmware-20240111-121.gitb3132c18.el8.noarch 1/2 2024-04-25T03:06:48.430 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : linux-firmware-20230217-113.git83f1d778.el8.noarch 2/2 2024-04-25T03:06:48.430 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:06:48.431 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-04-25T03:06:48.431 INFO:teuthology.orchestra.run.smithi188.stdout: linux-firmware-20240111-121.gitb3132c18.el8.noarch 2024-04-25T03:06:48.431 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:06:48.431 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-25T03:06:48.613 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on cluster2.mon.a... 2024-04-25T03:06:48.613 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-25T03:06:48.757 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/8/flavors/default/x86_64/ 2024-04-25T03:06:48.757 DEBUG:teuthology.orchestra.run.smithi188:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/8/flavors/default/x86_64/ --input-file=- 2024-04-25T03:06:48.760 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi188.front.sepia.ceph.com to complete... 2024-04-25T03:07:51.395 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25 03:07:51 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/8/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [425099368/425099368] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-25T03:07:51.396 INFO:teuthology.orchestra.run.smithi188.stderr:FINISHED --2024-04-25 03:07:51-- 2024-04-25T03:07:51.396 INFO:teuthology.orchestra.run.smithi188.stderr:Total wall clock time: 1m 3s 2024-04-25T03:07:51.396 INFO:teuthology.orchestra.run.smithi188.stderr:Downloaded: 1 files, 405M in 1m 2s (6.54 MB/s) 2024-04-25T03:07:51.399 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on cluster2.mon.a... 2024-04-25T03:07:51.400 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-25T03:07:53.251 INFO:teuthology.orchestra.run.smithi188.stdout:Verifying... ######################################## 2024-04-25T03:07:53.295 INFO:teuthology.orchestra.run.smithi188.stdout:Preparing... ######################################## 2024-04-25T03:07:53.405 INFO:teuthology.orchestra.run.smithi188.stdout:Updating / installing... 2024-04-25T03:07:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25 03:07:58 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/8/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [425099368/425099368] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-25T03:07:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:FINISHED --2024-04-25 03:07:58-- 2024-04-25T03:07:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:Total wall clock time: 1m 23s 2024-04-25T03:07:58.662 INFO:teuthology.orchestra.run.smithi173.stderr:Downloaded: 1 files, 405M in 1m 23s (4.89 MB/s) 2024-04-25T03:07:58.665 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on cluster1.mon.a... 2024-04-25T03:07:58.665 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-25T03:08:00.483 INFO:teuthology.orchestra.run.smithi173.stdout:Verifying... ######################################## 2024-04-25T03:08:00.527 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing... ######################################## 2024-04-25T03:08:00.636 INFO:teuthology.orchestra.run.smithi173.stdout:Updating / installing... 2024-04-25T03:08:35.984 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-25T03:08:40.062 INFO:teuthology.orchestra.run.smithi188.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-25T03:08:43.217 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-25T03:08:45.340 INFO:teuthology.orchestra.run.smithi173.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-25T03:09:00.531 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-25T03:09:00.531 DEBUG:teuthology.orchestra.run.smithi188:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-25T03:09:00.596 INFO:teuthology.orchestra.run.smithi188.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-25T03:09:00.597 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-25T03:09:00.598 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-25T03:09:00.599 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-25T03:09:00.600 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-25T03:09:00.601 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-25T03:09:00.602 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-25T03:09:00.603 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-25T03:09:00.604 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-25T03:09:00.605 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-25T03:09:00.606 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-25T03:09:00.607 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-25T03:09:00.608 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-25T03:09:00.609 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-25T03:09:00.610 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-25T03:09:00.611 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-25T03:09:00.612 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-25T03:09:00.613 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-25T03:09:00.614 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-25T03:09:00.615 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-25T03:09:00.616 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-25T03:09:00.617 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-25T03:09:00.618 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-25T03:09:00.619 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-25T03:09:00.620 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-25T03:09:00.621 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-25T03:09:00.622 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-25T03:09:00.623 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-25T03:09:00.624 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-25T03:09:00.625 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-25T03:09:00.626 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-25T03:09:00.627 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-25T03:09:00.628 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-25T03:09:00.629 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-25T03:09:00.630 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-25T03:09:00.631 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-25T03:09:00.632 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-25T03:09:00.633 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-25T03:09:00.634 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-25T03:09:00.635 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-25T03:09:00.636 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-25T03:09:00.637 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-25T03:09:00.638 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-25T03:09:00.639 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-25T03:09:00.640 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-25T03:09:00.641 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-25T03:09:00.642 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-25T03:09:00.643 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-25T03:09:00.644 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-25T03:09:00.645 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-25T03:09:00.646 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-25T03:09:00.647 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-25T03:09:00.648 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-25T03:09:00.649 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-25T03:09:00.650 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-25T03:09:00.651 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-25T03:09:00.652 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-25T03:09:00.653 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-25T03:09:00.654 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-25T03:09:00.655 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-25T03:09:00.656 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-25T03:09:00.657 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-25T03:09:00.658 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-25T03:09:00.659 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-25T03:09:00.660 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-25T03:09:00.661 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-25T03:09:00.662 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-25T03:09:00.663 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-25T03:09:00.664 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-25T03:09:00.665 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-25T03:09:00.666 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-25T03:09:00.667 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-25T03:09:00.668 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-25T03:09:00.669 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-25T03:09:00.670 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-25T03:09:00.671 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-25T03:09:00.672 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-25T03:09:00.673 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-25T03:09:00.674 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-25T03:09:00.675 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-25T03:09:00.676 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-25T03:09:00.677 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-25T03:09:00.678 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-25T03:09:00.679 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-25T03:09:00.680 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-25T03:09:00.681 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-25T03:09:00.682 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-25T03:09:00.683 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-25T03:09:00.684 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-25T03:09:00.685 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-25T03:09:00.686 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-25T03:09:00.687 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-25T03:09:00.688 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-25T03:09:00.689 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-25T03:09:00.690 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-25T03:09:00.691 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-25T03:09:00.692 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-25T03:09:00.693 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-25T03:09:00.694 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-25T03:09:00.695 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-25T03:09:00.696 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-25T03:09:00.697 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-25T03:09:00.698 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-25T03:09:00.699 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-25T03:09:00.700 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-25T03:09:00.701 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-25T03:09:00.702 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-25T03:09:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-25T03:09:00.704 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-25T03:09:00.705 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-25T03:09:00.706 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-25T03:09:00.707 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-25T03:09:00.708 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-25T03:09:00.709 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-25T03:09:00.710 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-25T03:09:00.711 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-25T03:09:00.712 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-25T03:09:00.713 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-25T03:09:00.714 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-25T03:09:00.715 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-25T03:09:00.716 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-25T03:09:00.717 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-25T03:09:00.718 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-25T03:09:00.719 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-25T03:09:00.720 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-25T03:09:00.721 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-25T03:09:00.722 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-25T03:09:00.723 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-25T03:09:00.724 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-25T03:09:00.725 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-25T03:09:00.726 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-25T03:09:00.727 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-25T03:09:00.728 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-25T03:09:00.729 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-25T03:09:00.730 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-25T03:09:00.731 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-25T03:09:00.732 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-25T03:09:00.733 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-25T03:09:00.734 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-25T03:09:00.735 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-25T03:09:00.736 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-25T03:09:00.736 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-25T03:09:00.736 INFO:teuthology.orchestra.run.smithi188.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T03:09:00.740 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:00.740 DEBUG:teuthology.orchestra.run.smithi188:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-25T03:09:00.765 INFO:teuthology.orchestra.run.smithi188.stdout:postinstall scriptlet (using /bin/sh): 2024-04-25T03:09:00.765 INFO:teuthology.orchestra.run.smithi188.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T03:09:00.766 INFO:teuthology.orchestra.run.smithi188.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T03:09:00.766 INFO:teuthology.orchestra.run.smithi188.stdout:fi 2024-04-25T03:09:00.766 INFO:teuthology.orchestra.run.smithi188.stdout:for file in vmlinuz System.map config; do 2024-04-25T03:09:00.766 INFO:teuthology.orchestra.run.smithi188.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout: fi 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:done 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:fi 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:fi 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T03:09:00.767 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:00.768 INFO:teuthology.orchestra.run.smithi188.stdout:fi 2024-04-25T03:09:00.768 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-04-25T03:09:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-04-25T03:09:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-04-25T03:09:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.02 2024-04-25T03:09:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 129.el8 2024-04-25T03:09:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-04-25T03:09:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-25T03:09:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:Group : System Environment/Base 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 9265586 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Relocations : (not relocatable) 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : CentOS BuildSystem 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:09:00.846 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-25T03:09:00.847 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-25T03:09:00.847 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-25T03:09:00.847 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-04-25T03:09:00.847 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:09:00.847 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-04-25T03:09:00.848 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-25T03:09:00.848 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:00.848 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-25T03:09:05.476 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-04-25T03:09:05.504 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-25T03:09:05.504 DEBUG:teuthology.orchestra.run.smithi173:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-25T03:09:05.565 INFO:teuthology.orchestra.run.smithi173.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:05.565 INFO:teuthology.orchestra.run.smithi173.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:05.565 INFO:teuthology.orchestra.run.smithi173.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-25T03:09:05.566 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-25T03:09:05.567 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-25T03:09:05.568 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-25T03:09:05.569 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-25T03:09:05.570 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-25T03:09:05.571 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-25T03:09:05.572 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-25T03:09:05.573 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-25T03:09:05.574 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-25T03:09:05.575 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-25T03:09:05.576 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-25T03:09:05.577 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-25T03:09:05.578 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-25T03:09:05.579 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-25T03:09:05.580 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-25T03:09:05.581 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-25T03:09:05.582 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-25T03:09:05.583 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-25T03:09:05.584 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-25T03:09:05.585 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-25T03:09:05.586 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-25T03:09:05.587 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-25T03:09:05.587 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-25T03:09:05.588 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-25T03:09:05.589 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-25T03:09:05.590 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-25T03:09:05.591 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-25T03:09:05.592 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-25T03:09:05.592 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-25T03:09:05.592 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-25T03:09:05.592 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-25T03:09:05.592 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-25T03:09:05.593 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-25T03:09:05.594 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-25T03:09:05.595 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-25T03:09:05.596 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-25T03:09:05.597 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-25T03:09:05.598 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-25T03:09:05.599 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-25T03:09:05.600 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-25T03:09:05.601 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-25T03:09:05.602 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-25T03:09:05.603 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-25T03:09:05.604 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-25T03:09:05.605 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-25T03:09:05.606 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-25T03:09:05.607 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-25T03:09:05.608 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-25T03:09:05.609 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-25T03:09:05.610 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-25T03:09:05.611 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-25T03:09:05.612 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-25T03:09:05.613 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-25T03:09:05.614 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-25T03:09:05.615 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-25T03:09:05.616 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-25T03:09:05.617 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-25T03:09:05.618 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-25T03:09:05.619 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-25T03:09:05.620 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-25T03:09:05.621 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-25T03:09:05.622 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-25T03:09:05.623 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-25T03:09:05.624 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-25T03:09:05.625 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-25T03:09:05.626 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-25T03:09:05.627 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-25T03:09:05.628 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-25T03:09:05.629 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-25T03:09:05.630 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-25T03:09:05.631 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-25T03:09:05.632 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-25T03:09:05.633 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-25T03:09:05.634 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-25T03:09:05.635 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-25T03:09:05.636 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-25T03:09:05.637 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-25T03:09:05.638 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-25T03:09:05.639 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-25T03:09:05.640 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-25T03:09:05.641 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-25T03:09:05.642 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-25T03:09:05.643 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-25T03:09:05.644 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-25T03:09:05.645 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-25T03:09:05.646 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-25T03:09:05.647 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-25T03:09:05.647 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-25T03:09:05.647 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-25T03:09:05.647 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-25T03:09:05.647 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-25T03:09:05.648 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-25T03:09:05.649 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-25T03:09:05.650 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-25T03:09:05.651 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-25T03:09:05.652 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-25T03:09:05.653 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-25T03:09:05.654 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-25T03:09:05.655 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-25T03:09:05.656 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-25T03:09:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-25T03:09:05.658 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-25T03:09:05.659 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-25T03:09:05.660 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-25T03:09:05.661 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-25T03:09:05.662 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-25T03:09:05.663 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-25T03:09:05.664 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-25T03:09:05.665 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-25T03:09:05.666 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-25T03:09:05.667 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-25T03:09:05.668 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-25T03:09:05.669 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-25T03:09:05.670 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-25T03:09:05.671 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-25T03:09:05.672 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-25T03:09:05.673 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-25T03:09:05.674 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-25T03:09:05.675 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-25T03:09:05.676 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-25T03:09:05.677 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-25T03:09:05.678 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-25T03:09:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-25T03:09:05.680 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-25T03:09:05.681 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-25T03:09:05.682 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-25T03:09:05.683 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-25T03:09:05.684 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-25T03:09:05.685 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-25T03:09:05.686 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-25T03:09:05.687 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-25T03:09:05.688 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-25T03:09:05.689 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-25T03:09:05.690 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-25T03:09:05.691 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-25T03:09:05.692 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-25T03:09:05.693 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-25T03:09:05.694 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-25T03:09:05.695 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-25T03:09:05.696 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-25T03:09:05.697 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-25T03:09:05.698 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-25T03:09:05.699 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-25T03:09:05.700 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-25T03:09:05.701 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-25T03:09:05.702 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-25T03:09:05.703 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-25T03:09:05.704 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-25T03:09:05.705 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-25T03:09:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-25T03:09:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-25T03:09:05.708 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-25T03:09:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-25T03:09:05.710 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-25T03:09:05.711 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-25T03:09:05.712 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-25T03:09:05.713 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-25T03:09:05.714 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-25T03:09:05.715 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-25T03:09:05.716 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-25T03:09:05.717 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-25T03:09:05.718 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-25T03:09:05.719 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-25T03:09:05.719 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-25T03:09:05.719 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-25T03:09:05.719 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-25T03:09:05.719 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-25T03:09:05.719 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-25T03:09:05.719 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T03:09:05.723 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:05.723 DEBUG:teuthology.orchestra.run.smithi173:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-25T03:09:05.747 INFO:teuthology.orchestra.run.smithi173.stdout:postinstall scriptlet (using /bin/sh): 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout:fi 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout:for file in vmlinuz System.map config; do 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout: fi 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout:done 2024-04-25T03:09:05.748 INFO:teuthology.orchestra.run.smithi173.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout:fi 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout:fi 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:05.749 INFO:teuthology.orchestra.run.smithi173.stdout:fi 2024-04-25T03:09:05.750 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.02 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 129.el8 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Group : System Environment/Base 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 9265586 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-04-25T03:09:05.824 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Relocations : (not relocatable) 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : CentOS BuildSystem 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-25T03:09:05.825 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-25T03:09:05.826 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-04-25T03:09:05.826 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:09:05.826 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-04-25T03:09:05.827 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-25T03:09:05.827 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:05.827 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-25T03:09:07.293 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-04-25T03:09:07.295 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-04-25T03:09:07.321 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.n0tH0SX97f 2024-04-25T03:09:07.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.n0tH0SX97f 2024-04-25T03:09:07.395 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.n0tH0SX97f 2024-04-25T03:09:07.535 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.n0tH0SX97f is 6KB 2024-04-25T03:09:07.583 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.n0tH0SX97f 2024-04-25T03:09:07.600 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-04-25T03:09:07.673 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-25T03:09:07.673 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-25T03:09:07.673 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-25T03:09:07.673 INFO:teuthology.orchestra.run.smithi188.stdout:a1b29d6c289747bdb22930598d39874a-6.9.0-rc3-g806ec36b2112.conf 2024-04-25T03:09:07.674 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:07.834 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-04-25T03:09:11.656 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-04-25T03:09:13.412 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-04-25T03:09:13.413 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-04-25T03:09:13.448 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.S3itOMDzb3 2024-04-25T03:09:13.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.S3itOMDzb3 2024-04-25T03:09:13.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.S3itOMDzb3 2024-04-25T03:09:13.690 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.S3itOMDzb3 is 6KB 2024-04-25T03:09:13.737 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.S3itOMDzb3 2024-04-25T03:09:13.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-04-25T03:09:13.819 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-25T03:09:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-25T03:09:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-25T03:09:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:a1b29d6c289747bdb22930598d39874a-6.9.0-rc3-g806ec36b2112.conf 2024-04-25T03:09:13.821 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-6.9.0-rc3-g806ec36b2112 2024-04-25T03:09:13.991 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-04-25T03:09:37.858 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-04-25T03:09:37.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-25T03:09:37.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:09:43.998 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-04-25T03:09:43.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T03:09:43.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:10:02.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-04-25T03:10:11.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T03:10:11.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:10:14.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-04-25T03:10:26.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T03:10:26.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:10:37.870 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T03:10:46.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-25T03:10:46.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:10:47.287 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-25T03:10:47.811 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-25T03:10:47.811 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "6.9.0-rc3-g806ec36b2112"... 2024-04-25T03:10:47.811 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-04-25T03:10:47.854 INFO:teuthology.orchestra.run.smithi188.stdout:6.9.0-rc3-g806ec36b2112 2024-04-25T03:10:47.854 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-25T03:10:47.854 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T03:10:47.855 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-25T03:10:48.855 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-04-25T03:10:48.855 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T03:10:48.960 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-04-25T03:10:49.001 DEBUG:teuthology.parallel:result is None 2024-04-25T03:10:58.549 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-25T03:10:59.114 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T03:10:59.114 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "6.9.0-rc3-g806ec36b2112"... 2024-04-25T03:10:59.115 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-04-25T03:10:59.157 INFO:teuthology.orchestra.run.smithi173.stdout:6.9.0-rc3-g806ec36b2112 2024-04-25T03:10:59.158 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-25T03:10:59.158 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T03:10:59.158 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-25T03:11:00.159 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-04-25T03:11:00.159 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T03:11:00.320 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-04-25T03:11:00.350 DEBUG:teuthology.parallel:result is None 2024-04-25T03:11:00.350 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T03:11:00.357 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T03:11:00.358 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T03:11:00.361 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T03:11:00.413 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T03:11:00.427 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T03:11:00.452 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T03:11:00.453 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T03:11:00.457 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T03:11:00.718 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T03:11:00.725 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T03:11:00.725 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T03:11:00.730 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T03:11:00.814 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T03:11:00.820 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T03:11:00.839 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T03:11:00.845 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T03:11:00.849 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T03:11:00.857 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T03:11:00.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T03:11:00.861 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T03:11:00.928 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T03:11:00.938 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T03:11:00.938 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T03:11:00.942 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T03:11:00.985 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T03:11:01.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T03:11:01.202 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T03:11:01.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T03:11:01.361 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:11:01.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T03:11:01.438 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T03:11:01.511 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T03:11:01.585 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T03:11:01.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T03:11:01.723 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:11:01.723 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T03:11:01.790 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-04-25T03:11:01.794 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-04-25T03:11:01.904 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T03:11:01.942 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T03:11:02.427 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T03:11:02.437 INFO:teuthology.task.internal:Starting timer... 2024-04-25T03:11:02.437 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T03:11:02.491 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T03:11:02.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-04-25T03:11:02.624 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-04-25T03:11:02.627 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-04-25T03:11:02.748 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-04-25T03:11:02.750 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T03:11:02.750 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-04-25T03:11:02.810 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-04-25T03:11:02.810 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-04-25T03:11:02.857 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-04-25T03:11:02.857 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi173.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-04-25T03:11:02.857 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-25T03:11:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T03:11:02.924 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-25T03:11:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T03:11:02.992 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T03:11:02.992 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-04-25T03:11:03.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-04-25T03:11:03.123 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T03:11:03.133 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T03:11:03.150 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T03:11:03.152 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi173.front.sepia.ceph.com,smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T03:18:14.469 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi173.front.sepia.ceph.com'), Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-04-25T03:18:14.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T03:18:14.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:18:14.561 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-25T03:18:14.678 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T03:18:14.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-25T03:18:14.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T03:18:14.770 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-04-25T03:18:14.890 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-04-25T03:18:14.891 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T03:18:14.904 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T03:18:14.904 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T03:18:14.905 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T03:18:14.909 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T03:18:14.909 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T03:18:14.977 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T03:18:15.003 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T03:18:15.008 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T03:18:15.035 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T03:18:15.103 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-04-25T03:18:15.128 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-04-25T03:18:15.129 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-04-25T03:18:15.155 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-04-25T03:18:15.158 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T03:18:15.186 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T03:18:15.189 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T03:18:15.217 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T03:18:15.273 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2024-04-25T03:18:15.280 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T03:18:15.280 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-25T03:18:15.280 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.281 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.281 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.281 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.305 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-04-25T03:18:15.311 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T03:18:15.312 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-25T03:18:15.312 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.312 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.312 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.312 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T03:18:15.313 INFO:teuthology.run_tasks:Running task install... 2024-04-25T03:18:15.322 DEBUG:teuthology.task.install:project ceph 2024-04-25T03:18:15.322 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}} 2024-04-25T03:18:15.323 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'} 2024-04-25T03:18:15.323 INFO:teuthology.task.install:Using flavor: default 2024-04-25T03:18:15.329 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T03:18:15.329 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-04-25T03:18:15.329 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'tag': None, 'wait_for_package': False} 2024-04-25T03:18:15.330 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-25T03:18:15.331 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'tag': None, 'wait_for_package': False} 2024-04-25T03:18:15.332 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-25T03:18:15.476 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/ 2024-04-25T03:18:15.476 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2424.gb98aab6c 2024-04-25T03:18:15.583 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/ 2024-04-25T03:18:15.584 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2424.gb98aab6c 2024-04-25T03:18:15.873 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T03:18:15.873 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:18:15.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T03:18:15.942 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror, pv, pv, pv on remote rpm x86_64 2024-04-25T03:18:15.942 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b98aab6c00f483888c049d0d8e7bc507e115dd86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T03:18:15.956 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/squid/b98aab6c00f483888c049d0d8e7bc507e115dd86/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T03:18:15.956 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:18:15.956 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T03:18:16.023 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror, pv, pv, pv, pv on remote rpm x86_64 2024-04-25T03:18:16.024 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b98aab6c00f483888c049d0d8e7bc507e115dd86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T03:18:16.051 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T03:18:16.132 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T03:18:16.142 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T03:18:16.208 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-04-25T03:18:16.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-04-25T03:18:16.222 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T03:18:16.288 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-04-25T03:18:16.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-04-25T03:18:16.553 INFO:teuthology.orchestra.run.smithi173.stdout:78 files removed 2024-04-25T03:18:16.591 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-mirror pv pv pv pv 2024-04-25T03:18:16.634 INFO:teuthology.orchestra.run.smithi188.stdout:78 files removed 2024-04-25T03:18:16.670 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-mirror pv pv pv pv 2024-04-25T03:18:18.296 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 75 kB/s | 86 kB 00:01 2024-04-25T03:18:18.306 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 80 kB/s | 86 kB 00:01 2024-04-25T03:18:18.973 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 18 kB/s | 11 kB 00:00 2024-04-25T03:18:19.279 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 12 kB/s | 11 kB 00:00 2024-04-25T03:18:19.649 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2024-04-25T03:18:20.080 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 1.9 kB/s | 1.5 kB 00:00 2024-04-25T03:18:20.416 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - AppStream 38 MB/s | 28 MB 00:00 2024-04-25T03:18:20.838 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - AppStream 38 MB/s | 28 MB 00:00 2024-04-25T03:18:26.541 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - BaseOS 13 MB/s | 10 MB 00:00 2024-04-25T03:18:27.173 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-04-25T03:18:28.449 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras 82 kB/s | 18 kB 00:00 2024-04-25T03:18:28.858 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 7.7 kB 00:00 2024-04-25T03:18:29.098 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras 74 kB/s | 18 kB 00:00 2024-04-25T03:18:29.374 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - Extras common packages 47 kB/s | 7.7 kB 00:00 2024-04-25T03:18:29.459 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 8 - PowerTools 9.8 MB/s | 5.0 MB 00:00 2024-04-25T03:18:30.134 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 8 - PowerTools 7.5 MB/s | 5.0 MB 00:00 2024-04-25T03:18:30.611 INFO:teuthology.orchestra.run.smithi188.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-04-25T03:18:31.269 INFO:teuthology.orchestra.run.smithi173.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-04-25T03:18:32.139 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-25T03:18:33.062 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 16 MB 00:01 2024-04-25T03:18:36.562 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 454 kB/s | 24 kB 00:00 2024-04-25T03:18:37.587 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 311 kB/s | 24 kB 00:00 2024-04-25T03:18:40.428 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-25T03:18:40.429 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-25T03:18:40.501 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 6.4 k 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 5.9 M 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 951 k 2024-04-25T03:18:40.521 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 174 k 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 1.7 M 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 141 k 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 3.5 M 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 7.4 M 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 49 k 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 15 M 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 56 M 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 263 k 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 486 k 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 32 k 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 869 k 2024-04-25T03:18:40.522 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 139 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: pv x86_64 1.6.6-7.el8 epel 64 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 197 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 372 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 364 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 107 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 92 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 3.6 M 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 180 k 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 3.8 M 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 3.8 M 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 25 M 2024-04-25T03:18:40.523 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 32 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 2.6 M 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 249 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 4.9 M 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 19 M 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 15 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 24 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 183 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-25T03:18:40.524 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 547 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 7.5 M 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.3.4-12.el8 powertools 30 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-25T03:18:40.525 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 45 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 133 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-25T03:18:40.526 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-25T03:18:40.527 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-25T03:18:40.528 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-25T03:18:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:18:40.530 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-04-25T03:18:40.530 INFO:teuthology.orchestra.run.smithi188.stdout:============================================================================================ 2024-04-25T03:18:40.530 INFO:teuthology.orchestra.run.smithi188.stdout:Install 105 Packages 2024-04-25T03:18:40.530 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2024-04-25T03:18:40.530 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:18:40.530 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 209 M 2024-04-25T03:18:40.530 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-04-25T03:18:41.336 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-25T03:18:41.337 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-04-25T03:18:41.387 INFO:teuthology.orchestra.run.smithi188.stdout:(1/107): ceph-19.0.0-2424.gb98aab6c.el8.x86_64. 16 kB/s | 6.4 kB 00:00 2024-04-25T03:18:41.409 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-25T03:18:41.426 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-04-25T03:18:41.426 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-04-25T03:18:41.426 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-04-25T03:18:41.426 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-04-25T03:18:41.426 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 6.4 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 5.9 M 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 951 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 174 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 1.7 M 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 141 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 3.5 M 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 7.4 M 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 49 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 15 M 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 56 M 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 263 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 486 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 32 k 2024-04-25T03:18:41.427 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 869 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 139 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: pv x86_64 1.6.6-7.el8 epel 64 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 197 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 372 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 364 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 107 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 92 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 3.6 M 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 180 k 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 3.8 M 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 3.8 M 2024-04-25T03:18:41.428 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 25 M 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 32 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 2.6 M 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 249 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 4.9 M 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 19 M 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2424.gb98aab6c.el8 ceph-noarch 15 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 24 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 183 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-04-25T03:18:41.429 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 547 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 7.5 M 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.3.4-12.el8 powertools 30 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.5.0-1.el8 epel 167 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-04-25T03:18:41.430 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 45 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-2424.gb98aab6c.el8 ceph 133 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-04-25T03:18:41.431 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-04-25T03:18:41.432 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 16.8-9.el8 powertools 48 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-25T03:18:41.433 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-04-25T03:18:41.434 INFO:teuthology.orchestra.run.smithi173.stdout:Install 105 Packages 2024-04-25T03:18:41.435 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-04-25T03:18:41.435 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:18:41.435 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 209 M 2024-04-25T03:18:41.435 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-04-25T03:18:42.624 INFO:teuthology.orchestra.run.smithi173.stdout:(1/107): ceph-19.0.0-2424.gb98aab6c.el8.x86_64. 16 kB/s | 6.4 kB 00:00 2024-04-25T03:18:43.379 INFO:teuthology.orchestra.run.smithi188.stdout:(2/107): ceph-fuse-19.0.0-2424.gb98aab6c.el8.x8 478 kB/s | 951 kB 00:01 2024-04-25T03:18:43.679 INFO:teuthology.orchestra.run.smithi188.stdout:(3/107): ceph-immutable-object-cache-19.0.0-242 580 kB/s | 174 kB 00:00 2024-04-25T03:18:44.616 INFO:teuthology.orchestra.run.smithi173.stdout:(2/107): ceph-fuse-19.0.0-2424.gb98aab6c.el8.x8 478 kB/s | 951 kB 00:01 2024-04-25T03:18:44.916 INFO:teuthology.orchestra.run.smithi173.stdout:(3/107): ceph-immutable-object-cache-19.0.0-242 580 kB/s | 174 kB 00:00 2024-04-25T03:18:45.513 INFO:teuthology.orchestra.run.smithi188.stdout:(4/107): ceph-mds-19.0.0-2424.gb98aab6c.el8.x86 1.4 MB/s | 2.6 MB 00:01 2024-04-25T03:18:46.205 INFO:teuthology.orchestra.run.smithi188.stdout:(5/107): ceph-mgr-19.0.0-2424.gb98aab6c.el8.x86 2.4 MB/s | 1.7 MB 00:00 2024-04-25T03:18:47.083 INFO:teuthology.orchestra.run.smithi173.stdout:(4/107): ceph-mds-19.0.0-2424.gb98aab6c.el8.x86 1.2 MB/s | 2.6 MB 00:02 2024-04-25T03:18:48.446 INFO:teuthology.orchestra.run.smithi188.stdout:(6/107): ceph-mon-19.0.0-2424.gb98aab6c.el8.x86 2.2 MB/s | 4.9 MB 00:02 2024-04-25T03:18:48.641 INFO:teuthology.orchestra.run.smithi173.stdout:(5/107): ceph-mgr-19.0.0-2424.gb98aab6c.el8.x86 1.1 MB/s | 1.7 MB 00:01 2024-04-25T03:18:52.683 INFO:teuthology.orchestra.run.smithi173.stdout:(6/107): ceph-mon-19.0.0-2424.gb98aab6c.el8.x86 1.2 MB/s | 4.9 MB 00:04 2024-04-25T03:18:53.199 INFO:teuthology.orchestra.run.smithi173.stdout:(7/107): ceph-base-19.0.0-2424.gb98aab6c.el8.x8 548 kB/s | 5.9 MB 00:10 2024-04-25T03:18:55.153 INFO:teuthology.orchestra.run.smithi188.stdout:(7/107): ceph-base-19.0.0-2424.gb98aab6c.el8.x8 424 kB/s | 5.9 MB 00:14 2024-04-25T03:18:58.770 INFO:teuthology.orchestra.run.smithi188.stdout:(8/107): ceph-osd-19.0.0-2424.gb98aab6c.el8.x86 1.8 MB/s | 19 MB 00:10 2024-04-25T03:18:58.903 INFO:teuthology.orchestra.run.smithi188.stdout:(9/107): ceph-selinux-19.0.0-2424.gb98aab6c.el8 183 kB/s | 24 kB 00:00 2024-04-25T03:19:02.224 INFO:teuthology.orchestra.run.smithi173.stdout:(8/107): ceph-radosgw-19.0.0-2424.gb98aab6c.el8 1.7 MB/s | 15 MB 00:09 2024-04-25T03:19:02.324 INFO:teuthology.orchestra.run.smithi173.stdout:(9/107): ceph-selinux-19.0.0-2424.gb98aab6c.el8 245 kB/s | 24 kB 00:00 2024-04-25T03:19:03.366 INFO:teuthology.orchestra.run.smithi173.stdout:(10/107): ceph-common-19.0.0-2424.gb98aab6c.el8 1.2 MB/s | 25 MB 00:21 2024-04-25T03:19:03.491 INFO:teuthology.orchestra.run.smithi173.stdout:(11/107): libcephfs-devel-19.0.0-2424.gb98aab6c 254 kB/s | 32 kB 00:00 2024-04-25T03:19:04.500 INFO:teuthology.orchestra.run.smithi173.stdout:(12/107): libcephfs2-19.0.0-2424.gb98aab6c.el8. 862 kB/s | 869 kB 00:01 2024-04-25T03:19:04.783 INFO:teuthology.orchestra.run.smithi173.stdout:(13/107): libcephsqlite-19.0.0-2424.gb98aab6c.e 649 kB/s | 183 kB 00:00 2024-04-25T03:19:05.000 INFO:teuthology.orchestra.run.smithi173.stdout:(14/107): librados-devel-19.0.0-2424.gb98aab6c. 642 kB/s | 139 kB 00:00 2024-04-25T03:19:05.592 INFO:teuthology.orchestra.run.smithi173.stdout:(15/107): libradosstriper1-19.0.0-2424.gb98aab6 926 kB/s | 547 kB 00:00 2024-04-25T03:19:08.576 INFO:teuthology.orchestra.run.smithi173.stdout:(16/107): ceph-osd-19.0.0-2424.gb98aab6c.el8.x8 1.2 MB/s | 19 MB 00:15 2024-04-25T03:19:08.701 INFO:teuthology.orchestra.run.smithi173.stdout:(17/107): python3-ceph-argparse-19.0.0-2424.gb9 359 kB/s | 45 kB 00:00 2024-04-25T03:19:08.918 INFO:teuthology.orchestra.run.smithi173.stdout:(18/107): python3-ceph-common-19.0.0-2424.gb98a 614 kB/s | 133 kB 00:00 2024-04-25T03:19:09.126 INFO:teuthology.orchestra.run.smithi173.stdout:(19/107): python3-cephfs-19.0.0-2424.gb98aab6c. 949 kB/s | 197 kB 00:00 2024-04-25T03:19:09.443 INFO:teuthology.orchestra.run.smithi173.stdout:(20/107): python3-rados-19.0.0-2424.gb98aab6c.e 1.1 MB/s | 372 kB 00:00 2024-04-25T03:19:09.760 INFO:teuthology.orchestra.run.smithi173.stdout:(21/107): python3-rbd-19.0.0-2424.gb98aab6c.el8 1.1 MB/s | 364 kB 00:00 2024-04-25T03:19:09.869 INFO:teuthology.orchestra.run.smithi173.stdout:(22/107): python3-rgw-19.0.0-2424.gb98aab6c.el8 986 kB/s | 107 kB 00:00 2024-04-25T03:19:10.003 INFO:teuthology.orchestra.run.smithi173.stdout:(23/107): rbd-fuse-19.0.0-2424.gb98aab6c.el8.x8 693 kB/s | 92 kB 00:00 2024-04-25T03:19:12.661 INFO:teuthology.orchestra.run.smithi173.stdout:(24/107): librgw2-19.0.0-2424.gb98aab6c.el8.x86 1.1 MB/s | 7.5 MB 00:07 2024-04-25T03:19:12.944 INFO:teuthology.orchestra.run.smithi173.stdout:(25/107): rbd-nbd-19.0.0-2424.gb98aab6c.el8.x86 637 kB/s | 180 kB 00:00 2024-04-25T03:19:12.953 INFO:teuthology.orchestra.run.smithi188.stdout:(10/107): ceph-common-19.0.0-2424.gb98aab6c.el8 792 kB/s | 25 MB 00:31 2024-04-25T03:19:13.053 INFO:teuthology.orchestra.run.smithi173.stdout:(26/107): ceph-grafana-dashboards-19.0.0-2424.g 293 kB/s | 32 kB 00:00 2024-04-25T03:19:13.053 INFO:teuthology.orchestra.run.smithi188.stdout:(11/107): libcephfs-devel-19.0.0-2424.gb98aab6c 318 kB/s | 32 kB 00:00 2024-04-25T03:19:13.253 INFO:teuthology.orchestra.run.smithi173.stdout:(27/107): ceph-mgr-cephadm-19.0.0-2424.gb98aab6 706 kB/s | 141 kB 00:00 2024-04-25T03:19:13.695 INFO:teuthology.orchestra.run.smithi173.stdout:(28/107): rbd-mirror-19.0.0-2424.gb98aab6c.el8. 1.0 MB/s | 3.6 MB 00:03 2024-04-25T03:19:14.117 INFO:teuthology.orchestra.run.smithi188.stdout:(12/107): libcephfs2-19.0.0-2424.gb98aab6c.el8. 817 kB/s | 869 kB 00:01 2024-04-25T03:19:14.396 INFO:teuthology.orchestra.run.smithi188.stdout:(13/107): libcephsqlite-19.0.0-2424.gb98aab6c.e 658 kB/s | 183 kB 00:00 2024-04-25T03:19:14.597 INFO:teuthology.orchestra.run.smithi188.stdout:(14/107): ceph-radosgw-19.0.0-2424.gb98aab6c.el 801 kB/s | 15 MB 00:19 2024-04-25T03:19:14.696 INFO:teuthology.orchestra.run.smithi188.stdout:(15/107): librados-devel-19.0.0-2424.gb98aab6c. 463 kB/s | 139 kB 00:00 2024-04-25T03:19:15.555 INFO:teuthology.orchestra.run.smithi188.stdout:(16/107): libradosstriper1-19.0.0-2424.gb98aab6 572 kB/s | 547 kB 00:00 2024-04-25T03:19:15.680 INFO:teuthology.orchestra.run.smithi188.stdout:(17/107): python3-ceph-argparse-19.0.0-2424.gb9 358 kB/s | 45 kB 00:00 2024-04-25T03:19:16.029 INFO:teuthology.orchestra.run.smithi173.stdout:(29/107): ceph-mgr-dashboard-19.0.0-2424.gb98aa 1.3 MB/s | 3.5 MB 00:02 2024-04-25T03:19:16.089 INFO:teuthology.orchestra.run.smithi188.stdout:(18/107): python3-ceph-common-19.0.0-2424.gb98a 326 kB/s | 133 kB 00:00 2024-04-25T03:19:16.329 INFO:teuthology.orchestra.run.smithi173.stdout:(30/107): ceph-mgr-modules-core-19.0.0-2424.gb9 830 kB/s | 249 kB 00:00 2024-04-25T03:19:16.631 INFO:teuthology.orchestra.run.smithi188.stdout:(19/107): python3-cephfs-19.0.0-2424.gb98aab6c. 364 kB/s | 197 kB 00:00 2024-04-25T03:19:16.829 INFO:teuthology.orchestra.run.smithi173.stdout:(31/107): ceph-mgr-rook-19.0.0-2424.gb98aab6c.e 98 kB/s | 49 kB 00:00 2024-04-25T03:19:16.946 INFO:teuthology.orchestra.run.smithi173.stdout:(32/107): ceph-prometheus-alerts-19.0.0-2424.gb 130 kB/s | 15 kB 00:00 2024-04-25T03:19:17.338 INFO:teuthology.orchestra.run.smithi173.stdout:(33/107): ceph-volume-19.0.0-2424.gb98aab6c.el8 672 kB/s | 263 kB 00:00 2024-04-25T03:19:17.997 INFO:teuthology.orchestra.run.smithi173.stdout:(34/107): cephadm-19.0.0-2424.gb98aab6c.el8.noa 738 kB/s | 486 kB 00:00 2024-04-25T03:19:18.122 INFO:teuthology.orchestra.run.smithi173.stdout:(35/107): jq-1.6-8.el8.x86_64.rpm 1.6 MB/s | 203 kB 00:00 2024-04-25T03:19:18.214 INFO:teuthology.orchestra.run.smithi173.stdout:(36/107): librdkafka-1.6.1-1.el8.x86_64.rpm 7.2 MB/s | 677 kB 00:00 2024-04-25T03:19:18.217 INFO:teuthology.orchestra.run.smithi188.stdout:(20/107): python3-rados-19.0.0-2424.gb98aab6c.e 234 kB/s | 372 kB 00:01 2024-04-25T03:19:18.256 INFO:teuthology.orchestra.run.smithi173.stdout:(37/107): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2024-04-25T03:19:18.298 INFO:teuthology.orchestra.run.smithi173.stdout:(38/107): oniguruma-6.8.2-3.el8.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-04-25T03:19:18.432 INFO:teuthology.orchestra.run.smithi173.stdout:(39/107): openblas-0.3.15-4.el8.x86_64.rpm 36 MB/s | 4.7 MB 00:00 2024-04-25T03:19:18.549 INFO:teuthology.orchestra.run.smithi173.stdout:(40/107): openblas-threads-0.3.15-4.el8.x86_64. 42 MB/s | 4.9 MB 00:00 2024-04-25T03:19:18.674 INFO:teuthology.orchestra.run.smithi173.stdout:(41/107): python3-babel-2.5.1-7.el8.noarch.rpm 38 MB/s | 4.8 MB 00:00 2024-04-25T03:19:18.724 INFO:teuthology.orchestra.run.smithi173.stdout:(42/107): python3-jinja2-2.10.1-4.el8.noarch.rp 11 MB/s | 578 kB 00:00 2024-04-25T03:19:18.766 INFO:teuthology.orchestra.run.smithi173.stdout:(43/107): python3-jsonpatch-1.21-2.el8.noarch.r 639 kB/s | 27 kB 00:00 2024-04-25T03:19:18.808 INFO:teuthology.orchestra.run.smithi173.stdout:(44/107): python3-jsonpointer-1.10-11.el8.noarc 484 kB/s | 20 kB 00:00 2024-04-25T03:19:18.882 INFO:teuthology.orchestra.run.smithi173.stdout:(45/107): python3-mako-1.0.6-14.el8.noarch.rpm 2.1 MB/s | 157 kB 00:00 2024-04-25T03:19:18.926 INFO:teuthology.orchestra.run.smithi173.stdout:(46/107): python3-markupsafe-0.23-19.el8.x86_64 899 kB/s | 39 kB 00:00 2024-04-25T03:19:19.035 INFO:teuthology.orchestra.run.smithi173.stdout:(47/107): python3-numpy-1.14.3-10.el8.x86_64.rp 34 MB/s | 3.7 MB 00:00 2024-04-25T03:19:19.076 INFO:teuthology.orchestra.run.smithi173.stdout:(48/107): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2024-04-25T03:19:19.081 INFO:teuthology.orchestra.run.smithi188.stdout:(21/107): python3-rbd-19.0.0-2424.gb98aab6c.el8 422 kB/s | 364 kB 00:00 2024-04-25T03:19:19.110 INFO:teuthology.orchestra.run.smithi173.stdout:(49/107): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2024-04-25T03:19:19.152 INFO:teuthology.orchestra.run.smithi173.stdout:(50/107): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2024-04-25T03:19:19.202 INFO:teuthology.orchestra.run.smithi173.stdout:(51/107): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2024-04-25T03:19:19.236 INFO:teuthology.orchestra.run.smithi173.stdout:(52/107): python3-pyasn1-modules-0.3.7-6.el8.no 3.3 MB/s | 110 kB 00:00 2024-04-25T03:19:19.278 INFO:teuthology.orchestra.run.smithi173.stdout:(53/107): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-04-25T03:19:19.348 INFO:teuthology.orchestra.run.smithi188.stdout:(22/107): python3-rgw-19.0.0-2424.gb98aab6c.el8 400 kB/s | 107 kB 00:00 2024-04-25T03:19:19.537 INFO:teuthology.orchestra.run.smithi173.stdout:(54/107): python3-scipy-1.0.0-21.module_el8+762 55 MB/s | 14 MB 00:00 2024-04-25T03:19:19.548 INFO:teuthology.orchestra.run.smithi188.stdout:(23/107): rbd-fuse-19.0.0-2424.gb98aab6c.el8.x8 462 kB/s | 92 kB 00:00 2024-04-25T03:19:19.578 INFO:teuthology.orchestra.run.smithi173.stdout:(55/107): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2024-04-25T03:19:19.621 INFO:teuthology.orchestra.run.smithi173.stdout:(56/107): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2024-04-25T03:19:19.654 INFO:teuthology.orchestra.run.smithi173.stdout:(57/107): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.1 MB/s | 71 kB 00:00 2024-04-25T03:19:19.796 INFO:teuthology.orchestra.run.smithi173.stdout:(58/107): fuse-2.9.7-19.el8.x86_64.rpm 587 kB/s | 83 kB 00:00 2024-04-25T03:19:19.977 INFO:teuthology.orchestra.run.smithi173.stdout:(59/107): libgfortran-8.5.0-21.el8.x86_64.rpm 3.5 MB/s | 645 kB 00:00 2024-04-25T03:19:20.080 INFO:teuthology.orchestra.run.smithi173.stdout:(60/107): libquadmath-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2024-04-25T03:19:20.180 INFO:teuthology.orchestra.run.smithi173.stdout:(61/107): librabbitmq-0.9.0-5.el8.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-04-25T03:19:20.289 INFO:teuthology.orchestra.run.smithi173.stdout:(62/107): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2024-04-25T03:19:20.431 INFO:teuthology.orchestra.run.smithi173.stdout:(63/107): python3-cryptography-3.2.1-7.el8.x86_ 4.4 MB/s | 641 kB 00:00 2024-04-25T03:19:20.515 INFO:teuthology.orchestra.run.smithi173.stdout:(64/107): python3-jwt-1.6.1-2.el8.noarch.rpm 512 kB/s | 43 kB 00:00 2024-04-25T03:19:20.615 INFO:teuthology.orchestra.run.smithi173.stdout:(65/107): python3-oauthlib-2.1.0-1.el8.noarch.r 1.5 MB/s | 155 kB 00:00 2024-04-25T03:19:20.707 INFO:teuthology.orchestra.run.smithi173.stdout:(66/107): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2024-04-25T03:19:20.782 INFO:teuthology.orchestra.run.smithi173.stdout:(67/107): python3-requests-oauthlib-1.0.0-1.el8 567 kB/s | 43 kB 00:00 2024-04-25T03:19:20.899 INFO:teuthology.orchestra.run.smithi173.stdout:(68/107): lua-devel-5.3.4-12.el8.x86_64.rpm 257 kB/s | 30 kB 00:00 2024-04-25T03:19:21.008 INFO:teuthology.orchestra.run.smithi173.stdout:(69/107): python3-packaging-16.8-9.el8.noarch.r 445 kB/s | 48 kB 00:00 2024-04-25T03:19:21.317 INFO:teuthology.orchestra.run.smithi173.stdout:(70/107): gperftools-libs-2.7-9.el8.x86_64.rpm 995 kB/s | 306 kB 00:00 2024-04-25T03:19:21.375 INFO:teuthology.orchestra.run.smithi173.stdout:(71/107): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-04-25T03:19:21.434 INFO:teuthology.orchestra.run.smithi173.stdout:(72/107): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2024-04-25T03:19:21.493 INFO:teuthology.orchestra.run.smithi173.stdout:(73/107): luarocks-3.5.0-1.el8.noarch.rpm 2.8 MB/s | 167 kB 00:00 2024-04-25T03:19:21.543 INFO:teuthology.orchestra.run.smithi173.stdout:(74/107): pv-1.6.6-7.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T03:19:21.635 INFO:teuthology.orchestra.run.smithi173.stdout:(75/107): python3-asyncssh-2.7.0-2.el8.noarch.r 5.1 MB/s | 476 kB 00:00 2024-04-25T03:19:21.685 INFO:teuthology.orchestra.run.smithi173.stdout:(76/107): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 875 kB/s | 44 kB 00:00 2024-04-25T03:19:21.744 INFO:teuthology.orchestra.run.smithi173.stdout:(77/107): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2024-04-25T03:19:21.794 INFO:teuthology.orchestra.run.smithi173.stdout:(78/107): python3-cachetools-3.1.1-4.el8.noarch 659 kB/s | 33 kB 00:00 2024-04-25T03:19:21.845 INFO:teuthology.orchestra.run.smithi173.stdout:(79/107): python3-certifi-2018.10.15-7.el8.noar 326 kB/s | 16 kB 00:00 2024-04-25T03:19:21.895 INFO:teuthology.orchestra.run.smithi173.stdout:(80/107): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2024-04-25T03:19:21.962 INFO:teuthology.orchestra.run.smithi173.stdout:(81/107): python3-cherrypy-18.4.0-1.el8.noarch. 5.6 MB/s | 384 kB 00:00 2024-04-25T03:19:22.037 INFO:teuthology.orchestra.run.smithi173.stdout:(82/107): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2024-04-25T03:19:22.152 INFO:teuthology.orchestra.run.smithi173.stdout:(83/107): python3-jaraco-6.2-6.el8.noarch.rpm 97 kB/s | 11 kB 00:00 2024-04-25T03:19:22.229 INFO:teuthology.orchestra.run.smithi173.stdout:(84/107): python3-jaraco-functools-2.0-4.el8.no 227 kB/s | 18 kB 00:00 2024-04-25T03:19:22.472 INFO:teuthology.orchestra.run.smithi173.stdout:(85/107): python3-kubernetes-11.0.0-6.el8.noarc 4.8 MB/s | 1.2 MB 00:00 2024-04-25T03:19:22.539 INFO:teuthology.orchestra.run.smithi188.stdout:(24/107): librgw2-19.0.0-2424.gb98aab6c.el8.x86 985 kB/s | 7.5 MB 00:07 2024-04-25T03:19:22.563 INFO:teuthology.orchestra.run.smithi173.stdout:(86/107): python3-logutils-0.3.5-11.el8.noarch. 532 kB/s | 49 kB 00:00 2024-04-25T03:19:22.664 INFO:teuthology.orchestra.run.smithi173.stdout:(87/107): python3-more-itertools-7.2.0-3.el8.no 587 kB/s | 59 kB 00:00 2024-04-25T03:19:22.773 INFO:teuthology.orchestra.run.smithi188.stdout:(25/107): rbd-nbd-19.0.0-2424.gb98aab6c.el8.x86 775 kB/s | 180 kB 00:00 2024-04-25T03:19:22.781 INFO:teuthology.orchestra.run.smithi173.stdout:(88/107): python3-natsort-7.1.1-2.el8.noarch.rp 517 kB/s | 60 kB 00:00 2024-04-25T03:19:22.873 INFO:teuthology.orchestra.run.smithi188.stdout:(26/107): ceph-grafana-dashboards-19.0.0-2424.g 316 kB/s | 32 kB 00:00 2024-04-25T03:19:22.889 INFO:teuthology.orchestra.run.smithi173.stdout:(89/107): python3-pecan-1.3.2-9.el8.noarch.rpm 2.6 MB/s | 283 kB 00:00 2024-04-25T03:19:22.966 INFO:teuthology.orchestra.run.smithi173.stdout:(90/107): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2024-04-25T03:19:23.048 INFO:teuthology.orchestra.run.smithi173.stdout:(91/107): python3-repoze-lru-0.7-6.el8.noarch.r 405 kB/s | 34 kB 00:00 2024-04-25T03:19:23.049 INFO:teuthology.orchestra.run.smithi188.stdout:(27/107): ceph-mgr-cephadm-19.0.0-2424.gb98aab6 807 kB/s | 141 kB 00:00 2024-04-25T03:19:23.132 INFO:teuthology.orchestra.run.smithi173.stdout:(92/107): python3-routes-2.4.1-12.el8.noarch.rp 2.3 MB/s | 196 kB 00:00 2024-04-25T03:19:23.249 INFO:teuthology.orchestra.run.smithi173.stdout:(93/107): python3-rsa-4.9-2.el8.noarch.rpm 552 kB/s | 64 kB 00:00 2024-04-25T03:19:23.341 INFO:teuthology.orchestra.run.smithi173.stdout:(94/107): python3-simplegeneric-0.8.1-17.el8.no 204 kB/s | 19 kB 00:00 2024-04-25T03:19:23.441 INFO:teuthology.orchestra.run.smithi173.stdout:(95/107): python3-singledispatch-3.4.0.3-18.el8 239 kB/s | 24 kB 00:00 2024-04-25T03:19:23.533 INFO:teuthology.orchestra.run.smithi173.stdout:(96/107): python3-tempora-1.14.1-5.el8.noarch.r 319 kB/s | 29 kB 00:00 2024-04-25T03:19:23.642 INFO:teuthology.orchestra.run.smithi173.stdout:(97/107): python3-trustme-0.6.0-4.el8.noarch.rp 254 kB/s | 27 kB 00:00 2024-04-25T03:19:23.734 INFO:teuthology.orchestra.run.smithi173.stdout:(98/107): ceph-mgr-diskprediction-local-19.0.0- 755 kB/s | 7.4 MB 00:10 2024-04-25T03:19:23.759 INFO:teuthology.orchestra.run.smithi173.stdout:(99/107): python3-waitress-1.4.3-1.el8.noarch.r 2.1 MB/s | 255 kB 00:00 2024-04-25T03:19:23.859 INFO:teuthology.orchestra.run.smithi173.stdout:(100/107): python3-websocket-client-0.56.0-5.el 613 kB/s | 61 kB 00:00 2024-04-25T03:19:23.943 INFO:teuthology.orchestra.run.smithi173.stdout:(101/107): python3-webtest-2.0.33-1.el8.noarch. 1.0 MB/s | 85 kB 00:00 2024-04-25T03:19:24.027 INFO:teuthology.orchestra.run.smithi173.stdout:(102/107): python3-zc-lockfile-2.0-2.el8.noarch 276 kB/s | 23 kB 00:00 2024-04-25T03:19:24.244 INFO:teuthology.orchestra.run.smithi173.stdout:(103/107): python3-webob-1.8.5-1.el8.1.noarch.r 494 kB/s | 251 kB 00:00 2024-04-25T03:19:24.286 INFO:teuthology.orchestra.run.smithi173.stdout:(104/107): thrift-0.13.0-2.el8.x86_64.rpm 6.7 MB/s | 1.7 MB 00:00 2024-04-25T03:19:25.757 INFO:teuthology.orchestra.run.smithi188.stdout:(28/107): ceph-mgr-dashboard-19.0.0-2424.gb98aa 1.3 MB/s | 3.5 MB 00:02 2024-04-25T03:19:25.832 INFO:teuthology.orchestra.run.smithi188.stdout:(29/107): rbd-mirror-19.0.0-2424.gb98aab6c.el8. 591 kB/s | 3.6 MB 00:06 2024-04-25T03:19:26.565 INFO:teuthology.orchestra.run.smithi188.stdout:(30/107): ceph-mgr-modules-core-19.0.0-2424.gb9 339 kB/s | 249 kB 00:00 2024-04-25T03:19:26.782 INFO:teuthology.orchestra.run.smithi188.stdout:(31/107): ceph-mgr-rook-19.0.0-2424.gb98aab6c.e 227 kB/s | 49 kB 00:00 2024-04-25T03:19:26.908 INFO:teuthology.orchestra.run.smithi188.stdout:(32/107): ceph-prometheus-alerts-19.0.0-2424.gb 122 kB/s | 15 kB 00:00 2024-04-25T03:19:27.633 INFO:teuthology.orchestra.run.smithi188.stdout:(33/107): ceph-volume-19.0.0-2424.gb98aab6c.el8 363 kB/s | 263 kB 00:00 2024-04-25T03:19:28.892 INFO:teuthology.orchestra.run.smithi188.stdout:(34/107): cephadm-19.0.0-2424.gb98aab6c.el8.noa 386 kB/s | 486 kB 00:01 2024-04-25T03:19:29.017 INFO:teuthology.orchestra.run.smithi188.stdout:(35/107): jq-1.6-8.el8.x86_64.rpm 1.6 MB/s | 203 kB 00:00 2024-04-25T03:19:29.093 INFO:teuthology.orchestra.run.smithi188.stdout:(36/107): librdkafka-1.6.1-1.el8.x86_64.rpm 8.8 MB/s | 677 kB 00:00 2024-04-25T03:19:29.134 INFO:teuthology.orchestra.run.smithi188.stdout:(37/107): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2024-04-25T03:19:29.176 INFO:teuthology.orchestra.run.smithi188.stdout:(38/107): oniguruma-6.8.2-3.el8.x86_64.rpm 4.4 MB/s | 188 kB 00:00 2024-04-25T03:19:29.318 INFO:teuthology.orchestra.run.smithi188.stdout:(39/107): openblas-0.3.15-4.el8.x86_64.rpm 33 MB/s | 4.7 MB 00:00 2024-04-25T03:19:29.435 INFO:teuthology.orchestra.run.smithi188.stdout:(40/107): openblas-threads-0.3.15-4.el8.x86_64. 42 MB/s | 4.9 MB 00:00 2024-04-25T03:19:29.544 INFO:teuthology.orchestra.run.smithi188.stdout:(41/107): python3-babel-2.5.1-7.el8.noarch.rpm 44 MB/s | 4.8 MB 00:00 2024-04-25T03:19:29.594 INFO:teuthology.orchestra.run.smithi188.stdout:(42/107): python3-jinja2-2.10.1-4.el8.noarch.rp 11 MB/s | 578 kB 00:00 2024-04-25T03:19:29.627 INFO:teuthology.orchestra.run.smithi188.stdout:(43/107): python3-jsonpatch-1.21-2.el8.noarch.r 803 kB/s | 27 kB 00:00 2024-04-25T03:19:29.670 INFO:teuthology.orchestra.run.smithi188.stdout:(44/107): python3-jsonpointer-1.10-11.el8.noarc 478 kB/s | 20 kB 00:00 2024-04-25T03:19:29.703 INFO:teuthology.orchestra.run.smithi188.stdout:(45/107): python3-mako-1.0.6-14.el8.noarch.rpm 4.6 MB/s | 157 kB 00:00 2024-04-25T03:19:29.745 INFO:teuthology.orchestra.run.smithi188.stdout:(46/107): python3-markupsafe-0.23-19.el8.x86_64 926 kB/s | 39 kB 00:00 2024-04-25T03:19:29.846 INFO:teuthology.orchestra.run.smithi188.stdout:(47/107): python3-numpy-1.14.3-10.el8.x86_64.rp 37 MB/s | 3.7 MB 00:00 2024-04-25T03:19:29.879 INFO:teuthology.orchestra.run.smithi188.stdout:(48/107): python3-numpy-f2py-1.14.3-10.el8.x86_ 6.6 MB/s | 225 kB 00:00 2024-04-25T03:19:29.921 INFO:teuthology.orchestra.run.smithi188.stdout:(49/107): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2024-04-25T03:19:29.963 INFO:teuthology.orchestra.run.smithi188.stdout:(50/107): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2024-04-25T03:19:30.005 INFO:teuthology.orchestra.run.smithi188.stdout:(51/107): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2024-04-25T03:19:30.047 INFO:teuthology.orchestra.run.smithi188.stdout:(52/107): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2024-04-25T03:19:30.089 INFO:teuthology.orchestra.run.smithi188.stdout:(53/107): python3-pytz-2017.2-11.el8.noarch.rpm 1.3 MB/s | 57 kB 00:00 2024-04-25T03:19:30.331 INFO:teuthology.orchestra.run.smithi188.stdout:(54/107): python3-scipy-1.0.0-21.module_el8+762 59 MB/s | 14 MB 00:00 2024-04-25T03:19:30.381 INFO:teuthology.orchestra.run.smithi188.stdout:(55/107): python3-werkzeug-0.12.2-4.el8.noarch. 9.0 MB/s | 457 kB 00:00 2024-04-25T03:19:30.423 INFO:teuthology.orchestra.run.smithi188.stdout:(56/107): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2024-04-25T03:19:30.465 INFO:teuthology.orchestra.run.smithi188.stdout:(57/107): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-04-25T03:19:30.616 INFO:teuthology.orchestra.run.smithi188.stdout:(58/107): ceph-mgr-diskprediction-local-19.0.0- 1.5 MB/s | 7.4 MB 00:04 2024-04-25T03:19:30.666 INFO:teuthology.orchestra.run.smithi188.stdout:(59/107): fuse-2.9.7-19.el8.x86_64.rpm 414 kB/s | 83 kB 00:00 2024-04-25T03:19:30.744 INFO:teuthology.orchestra.run.smithi173.stdout:(105/107): librados2-19.0.0-2424.gb98aab6c.el8. 604 kB/s | 3.8 MB 00:06 2024-04-25T03:19:30.800 INFO:teuthology.orchestra.run.smithi188.stdout:(60/107): libquadmath-8.5.0-21.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-04-25T03:19:30.892 INFO:teuthology.orchestra.run.smithi188.stdout:(61/107): librabbitmq-0.9.0-5.el8.x86_64.rpm 514 kB/s | 47 kB 00:00 2024-04-25T03:19:30.926 INFO:teuthology.orchestra.run.smithi188.stdout:(62/107): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2024-04-25T03:19:31.026 INFO:teuthology.orchestra.run.smithi188.stdout:(63/107): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.8 MB/s | 249 kB 00:00 2024-04-25T03:19:31.118 INFO:teuthology.orchestra.run.smithi188.stdout:(64/107): python3-cryptography-3.2.1-7.el8.x86_ 3.3 MB/s | 641 kB 00:00 2024-04-25T03:19:31.143 INFO:teuthology.orchestra.run.smithi188.stdout:(65/107): python3-jwt-1.6.1-2.el8.noarch.rpm 364 kB/s | 43 kB 00:00 2024-04-25T03:19:31.260 INFO:teuthology.orchestra.run.smithi188.stdout:(66/107): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2024-04-25T03:19:31.285 INFO:teuthology.orchestra.run.smithi188.stdout:(67/107): python3-pycparser-2.14-14.el8.noarch. 769 kB/s | 109 kB 00:00 2024-04-25T03:19:31.369 INFO:teuthology.orchestra.run.smithi188.stdout:(68/107): python3-requests-oauthlib-1.0.0-1.el8 393 kB/s | 43 kB 00:00 2024-04-25T03:19:31.486 INFO:teuthology.orchestra.run.smithi188.stdout:(69/107): lua-devel-5.3.4-12.el8.x86_64.rpm 150 kB/s | 30 kB 00:00 2024-04-25T03:19:31.611 INFO:teuthology.orchestra.run.smithi188.stdout:(70/107): python3-packaging-16.8-9.el8.noarch.r 199 kB/s | 48 kB 00:00 2024-04-25T03:19:31.820 INFO:teuthology.orchestra.run.smithi188.stdout:(71/107): liboath-2.6.2-3.el8.x86_64.rpm 282 kB/s | 59 kB 00:00 2024-04-25T03:19:31.862 INFO:teuthology.orchestra.run.smithi188.stdout:(72/107): gperftools-libs-2.7-9.el8.x86_64.rpm 816 kB/s | 306 kB 00:00 2024-04-25T03:19:32.171 INFO:teuthology.orchestra.run.smithi188.stdout:(73/107): libunwind-1.3.1-3.el8.x86_64.rpm 214 kB/s | 75 kB 00:00 2024-04-25T03:19:32.194 INFO:teuthology.orchestra.run.smithi173.stdout:(106/107): librbd1-19.0.0-2424.gb98aab6c.el8.x8 489 kB/s | 3.8 MB 00:07 2024-04-25T03:19:32.196 INFO:teuthology.orchestra.run.smithi188.stdout:(74/107): luarocks-3.5.0-1.el8.noarch.rpm 501 kB/s | 167 kB 00:00 2024-04-25T03:19:32.229 INFO:teuthology.orchestra.run.smithi188.stdout:(75/107): pv-1.6.6-7.el8.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T03:19:32.263 INFO:teuthology.orchestra.run.smithi188.stdout:(76/107): python3-asyncssh-2.7.0-2.el8.noarch.r 7.0 MB/s | 476 kB 00:00 2024-04-25T03:19:32.280 INFO:teuthology.orchestra.run.smithi188.stdout:(77/107): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 872 kB/s | 44 kB 00:00 2024-04-25T03:19:32.314 INFO:teuthology.orchestra.run.smithi188.stdout:(78/107): python3-beautifulsoup4-4.6.3-2.el8.1. 3.6 MB/s | 185 kB 00:00 2024-04-25T03:19:32.339 INFO:teuthology.orchestra.run.smithi188.stdout:(79/107): python3-cachetools-3.1.1-4.el8.noarch 566 kB/s | 33 kB 00:00 2024-04-25T03:19:32.364 INFO:teuthology.orchestra.run.smithi188.stdout:(80/107): python3-certifi-2018.10.15-7.el8.noar 325 kB/s | 16 kB 00:00 2024-04-25T03:19:32.398 INFO:teuthology.orchestra.run.smithi188.stdout:(81/107): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2024-04-25T03:19:32.423 INFO:teuthology.orchestra.run.smithi188.stdout:(82/107): python3-cherrypy-18.4.0-1.el8.noarch. 6.4 MB/s | 384 kB 00:00 2024-04-25T03:19:32.449 INFO:teuthology.orchestra.run.smithi188.stdout:(83/107): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2024-04-25T03:19:32.483 INFO:teuthology.orchestra.run.smithi188.stdout:(84/107): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2024-04-25T03:19:32.508 INFO:teuthology.orchestra.run.smithi188.stdout:(85/107): python3-jaraco-functools-2.0-4.el8.no 301 kB/s | 18 kB 00:00 2024-04-25T03:19:32.566 INFO:teuthology.orchestra.run.smithi188.stdout:(86/107): python3-kubernetes-11.0.0-6.el8.noarc 14 MB/s | 1.2 MB 00:00 2024-04-25T03:19:32.592 INFO:teuthology.orchestra.run.smithi188.stdout:(87/107): python3-logutils-0.3.5-11.el8.noarch. 583 kB/s | 49 kB 00:00 2024-04-25T03:19:32.617 INFO:teuthology.orchestra.run.smithi188.stdout:(88/107): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2024-04-25T03:19:32.642 INFO:teuthology.orchestra.run.smithi188.stdout:(89/107): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2024-04-25T03:19:32.668 INFO:teuthology.orchestra.run.smithi188.stdout:(90/107): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2024-04-25T03:19:32.693 INFO:teuthology.orchestra.run.smithi188.stdout:(91/107): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2024-04-25T03:19:32.718 INFO:teuthology.orchestra.run.smithi188.stdout:(92/107): python3-repoze-lru-0.7-6.el8.noarch.r 673 kB/s | 34 kB 00:00 2024-04-25T03:19:32.744 INFO:teuthology.orchestra.run.smithi188.stdout:(93/107): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2024-04-25T03:19:32.769 INFO:teuthology.orchestra.run.smithi188.stdout:(94/107): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2024-04-25T03:19:32.794 INFO:teuthology.orchestra.run.smithi188.stdout:(95/107): python3-simplegeneric-0.8.1-17.el8.no 374 kB/s | 19 kB 00:00 2024-04-25T03:19:32.820 INFO:teuthology.orchestra.run.smithi188.stdout:(96/107): python3-singledispatch-3.4.0.3-18.el8 476 kB/s | 24 kB 00:00 2024-04-25T03:19:32.845 INFO:teuthology.orchestra.run.smithi188.stdout:(97/107): python3-tempora-1.14.1-5.el8.noarch.r 582 kB/s | 29 kB 00:00 2024-04-25T03:19:32.870 INFO:teuthology.orchestra.run.smithi188.stdout:(98/107): python3-trustme-0.6.0-4.el8.noarch.rp 552 kB/s | 27 kB 00:00 2024-04-25T03:19:32.904 INFO:teuthology.orchestra.run.smithi188.stdout:(99/107): python3-waitress-1.4.3-1.el8.noarch.r 4.3 MB/s | 255 kB 00:00 2024-04-25T03:19:32.929 INFO:teuthology.orchestra.run.smithi188.stdout:(100/107): python3-webob-1.8.5-1.el8.1.noarch.r 4.2 MB/s | 251 kB 00:00 2024-04-25T03:19:32.954 INFO:teuthology.orchestra.run.smithi188.stdout:(101/107): python3-websocket-client-0.56.0-5.el 1.2 MB/s | 61 kB 00:00 2024-04-25T03:19:32.980 INFO:teuthology.orchestra.run.smithi188.stdout:(102/107): python3-webtest-2.0.33-1.el8.noarch. 1.7 MB/s | 85 kB 00:00 2024-04-25T03:19:33.013 INFO:teuthology.orchestra.run.smithi188.stdout:(103/107): python3-zc-lockfile-2.0-2.el8.noarch 394 kB/s | 23 kB 00:00 2024-04-25T03:19:33.114 INFO:teuthology.orchestra.run.smithi188.stdout:(104/107): thrift-0.13.0-2.el8.x86_64.rpm 13 MB/s | 1.7 MB 00:00 2024-04-25T03:19:35.872 INFO:teuthology.orchestra.run.smithi188.stdout:(105/107): librbd1-19.0.0-2424.gb98aab6c.el8.x8 1.4 MB/s | 3.8 MB 00:02 2024-04-25T03:19:38.135 INFO:teuthology.orchestra.run.smithi173.stdout:(107/107): ceph-test-19.0.0-2424.gb98aab6c.el8. 1.6 MB/s | 56 MB 00:35 2024-04-25T03:19:38.139 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-04-25T03:19:38.139 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.7 MB/s | 209 MB 00:56 2024-04-25T03:19:38.804 INFO:teuthology.orchestra.run.smithi188.stdout:(106/107): librados2-19.0.0-2424.gb98aab6c.el8. 678 kB/s | 3.8 MB 00:05 2024-04-25T03:19:39.074 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-04-25T03:19:39.181 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-04-25T03:19:39.181 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-04-25T03:19:40.504 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-04-25T03:19:40.508 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-04-25T03:19:44.717 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-04-25T03:19:44.914 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/109 2024-04-25T03:19:45.121 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/109 2024-04-25T03:19:45.317 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 3/109 2024-04-25T03:19:45.482 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/109 2024-04-25T03:19:45.958 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/109 2024-04-25T03:19:46.165 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 5/109 2024-04-25T03:19:46.617 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 5/109 2024-04-25T03:19:46.806 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 6/109 2024-04-25T03:19:46.947 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 6/109 2024-04-25T03:19:47.173 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 7/109 2024-04-25T03:19:47.285 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 7/109 2024-04-25T03:19:47.555 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-2424.gb98aab6c.el8.x86_64 8/109 2024-04-25T03:19:48.458 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/109 2024-04-25T03:19:48.664 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 10/109 2024-04-25T03:19:48.755 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 10/109 2024-04-25T03:19:48.998 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 11/109 2024-04-25T03:19:49.198 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/109 2024-04-25T03:19:49.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 13/109 2024-04-25T03:19:49.495 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 13/109 2024-04-25T03:19:49.648 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86 14/109 2024-04-25T03:19:49.777 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86 14/109 2024-04-25T03:19:49.968 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 15/109 2024-04-25T03:19:50.089 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 16/109 2024-04-25T03:19:50.258 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 16/109 2024-04-25T03:19:50.364 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 17/109 2024-04-25T03:19:51.046 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 17/109 2024-04-25T03:19:51.206 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.15-4.el8.x86_64 18/109 2024-04-25T03:19:51.882 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 18/109 2024-04-25T03:19:52.056 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 19/109 2024-04-25T03:19:52.877 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 19/109 2024-04-25T03:19:53.214 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/109 2024-04-25T03:19:53.389 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 21/109 2024-04-25T03:19:53.546 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 22/109 2024-04-25T03:19:53.805 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/109 2024-04-25T03:19:54.013 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-2424.gb98aab6c.el8. 24/109 2024-04-25T03:19:54.160 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-2424.gb98aab6c.el 25/109 2024-04-25T03:19:54.413 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-2424.gb98aab6c.el8.x86_6 26/109 2024-04-25T03:19:55.498 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 27/109 2024-04-25T03:19:55.861 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 28/109 2024-04-25T03:19:56.140 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 29/109 2024-04-25T03:19:56.330 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/109 2024-04-25T03:19:58.410 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/109 2024-04-25T03:19:58.709 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 31/109 2024-04-25T03:19:58.895 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 32/109 2024-04-25T03:19:59.067 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el8.noarch 33/109 2024-04-25T03:19:59.329 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 34/109 2024-04-25T03:19:59.533 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 35/109 2024-04-25T03:19:59.723 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-2424.gb98aab6c.el8.x86_6 36/109 2024-04-25T03:19:59.888 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 37/109 2024-04-25T03:20:00.144 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 38/109 2024-04-25T03:20:00.371 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 39/109 2024-04-25T03:20:00.518 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 40/109 2024-04-25T03:20:00.658 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 41/109 2024-04-25T03:20:00.837 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 42/109 2024-04-25T03:20:01.036 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 43/109 2024-04-25T03:20:01.218 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 44/109 2024-04-25T03:20:01.399 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 45/109 2024-04-25T03:20:01.544 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 46/109 2024-04-25T03:20:01.718 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 47/109 2024-04-25T03:20:01.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 48/109 2024-04-25T03:20:02.046 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-2.6-1.el8.noarch 49/109 2024-04-25T03:20:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 50/109 2024-04-25T03:20:02.490 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 51/109 2024-04-25T03:20:02.718 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 52/109 2024-04-25T03:20:02.911 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 53/109 2024-04-25T03:20:03.246 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 54/109 2024-04-25T03:20:03.588 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 55/109 2024-04-25T03:20:03.591 INFO:teuthology.orchestra.run.smithi188.stdout:(107/107): ceph-test-19.0.0-2424.gb98aab6c.el8. 882 kB/s | 56 MB 01:04 2024-04-25T03:20:03.594 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-04-25T03:20:03.594 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.5 MB/s | 209 MB 01:23 2024-04-25T03:20:03.782 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.5.0-1.el8.noarch 56/109 2024-04-25T03:20:03.984 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 57/109 2024-04-25T03:20:04.193 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 58/109 2024-04-25T03:20:04.404 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-16.8-9.el8.noarch 59/109 2024-04-25T03:20:04.518 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-04-25T03:20:04.611 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 60/109 2024-04-25T03:20:04.625 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-04-25T03:20:04.625 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-04-25T03:20:04.823 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 61/109 2024-04-25T03:20:05.225 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 62/109 2024-04-25T03:20:05.485 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 63/109 2024-04-25T03:20:05.693 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 64/109 2024-04-25T03:20:05.931 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 65/109 2024-04-25T03:20:05.943 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-04-25T03:20:05.947 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-04-25T03:20:06.138 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 66/109 2024-04-25T03:20:07.365 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 67/109 2024-04-25T03:20:07.710 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 68/109 2024-04-25T03:20:07.980 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 69/109 2024-04-25T03:20:08.207 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 70/109 2024-04-25T03:20:08.511 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 71/109 2024-04-25T03:20:08.729 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 72/109 2024-04-25T03:20:08.967 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 73/109 2024-04-25T03:20:09.473 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.7-19.el8.x86_64 74/109 2024-04-25T03:20:09.670 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 75/109 2024-04-25T03:20:09.854 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 76/109 2024-04-25T03:20:10.020 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 77/109 2024-04-25T03:20:10.183 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-04-25T03:20:10.378 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/109 2024-04-25T03:20:10.560 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/109 2024-04-25T03:20:10.758 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 3/109 2024-04-25T03:20:10.954 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/109 2024-04-25T03:20:11.438 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/109 2024-04-25T03:20:11.654 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 5/109 2024-04-25T03:20:12.124 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 5/109 2024-04-25T03:20:12.312 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 6/109 2024-04-25T03:20:12.462 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 6/109 2024-04-25T03:20:12.679 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 7/109 2024-04-25T03:20:12.776 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 7/109 2024-04-25T03:20:13.019 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:19.0.0-2424.gb98aab6c.el8.x86_64 8/109 2024-04-25T03:20:13.095 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 78/109 2024-04-25T03:20:13.345 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 78/109 2024-04-25T03:20:13.907 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 9/109 2024-04-25T03:20:14.060 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 78/109 2024-04-25T03:20:14.070 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 10/109 2024-04-25T03:20:14.161 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 10/109 2024-04-25T03:20:14.329 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 79/109 2024-04-25T03:20:14.386 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 11/109 2024-04-25T03:20:14.528 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 79/109 2024-04-25T03:20:14.620 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 12/109 2024-04-25T03:20:14.761 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 13/109 2024-04-25T03:20:14.785 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 80/109 2024-04-25T03:20:14.894 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 13/109 2024-04-25T03:20:15.070 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86 14/109 2024-04-25T03:20:15.218 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86 14/109 2024-04-25T03:20:15.494 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 15/109 2024-04-25T03:20:15.628 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 16/109 2024-04-25T03:20:15.773 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 16/109 2024-04-25T03:20:15.911 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 17/109 2024-04-25T03:20:16.571 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 17/109 2024-04-25T03:20:16.803 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.15-4.el8.x86_64 18/109 2024-04-25T03:20:17.463 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 18/109 2024-04-25T03:20:17.619 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 19/109 2024-04-25T03:20:18.434 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 19/109 2024-04-25T03:20:18.786 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/109 2024-04-25T03:20:18.961 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 21/109 2024-04-25T03:20:19.118 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 22/109 2024-04-25T03:20:19.342 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/109 2024-04-25T03:20:19.551 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:19.0.0-2424.gb98aab6c.el8. 24/109 2024-04-25T03:20:19.691 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:19.0.0-2424.gb98aab6c.el 25/109 2024-04-25T03:20:19.942 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:19.0.0-2424.gb98aab6c.el8.x86_6 26/109 2024-04-25T03:20:21.038 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 27/109 2024-04-25T03:20:21.415 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 28/109 2024-04-25T03:20:21.687 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 29/109 2024-04-25T03:20:21.868 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/109 2024-04-25T03:20:23.970 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 30/109 2024-04-25T03:20:24.288 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 31/109 2024-04-25T03:20:24.500 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 32/109 2024-04-25T03:20:24.705 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el8.noarch 33/109 2024-04-25T03:20:24.965 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 34/109 2024-04-25T03:20:25.167 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 35/109 2024-04-25T03:20:25.378 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:19.0.0-2424.gb98aab6c.el8.x86_6 36/109 2024-04-25T03:20:25.542 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 37/109 2024-04-25T03:20:25.790 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 38/109 2024-04-25T03:20:26.001 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 39/109 2024-04-25T03:20:26.173 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 40/109 2024-04-25T03:20:26.321 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 41/109 2024-04-25T03:20:26.488 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 42/109 2024-04-25T03:20:26.682 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 43/109 2024-04-25T03:20:26.872 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 44/109 2024-04-25T03:20:27.046 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 45/109 2024-04-25T03:20:27.190 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 46/109 2024-04-25T03:20:27.364 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 47/109 2024-04-25T03:20:27.565 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 48/109 2024-04-25T03:20:27.753 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-2.6-1.el8.noarch 49/109 2024-04-25T03:20:27.918 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 50/109 2024-04-25T03:20:28.131 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 51/109 2024-04-25T03:20:28.354 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 52/109 2024-04-25T03:20:28.555 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 53/109 2024-04-25T03:20:28.892 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 54/109 2024-04-25T03:20:29.219 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 55/109 2024-04-25T03:20:29.419 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.5.0-1.el8.noarch 56/109 2024-04-25T03:20:29.606 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 57/109 2024-04-25T03:20:29.814 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 58/109 2024-04-25T03:20:30.009 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-packaging-16.8-9.el8.noarch 59/109 2024-04-25T03:20:30.194 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.3.4-12.el8.x86_64 60/109 2024-04-25T03:20:30.418 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 61/109 2024-04-25T03:20:30.847 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 62/109 2024-04-25T03:20:31.106 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 63/109 2024-04-25T03:20:31.298 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 64/109 2024-04-25T03:20:31.519 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 65/109 2024-04-25T03:20:31.751 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 66/109 2024-04-25T03:20:32.989 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 67/109 2024-04-25T03:20:33.400 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 68/109 2024-04-25T03:20:33.867 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 69/109 2024-04-25T03:20:34.077 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 70/109 2024-04-25T03:20:34.407 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 71/109 2024-04-25T03:20:34.638 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 72/109 2024-04-25T03:20:34.861 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 73/109 2024-04-25T03:20:35.018 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.7-19.el8.x86_64 74/109 2024-04-25T03:20:35.218 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 75/109 2024-04-25T03:20:35.383 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 76/109 2024-04-25T03:20:35.532 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 77/109 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 80/109 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /run 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-04-25T03:20:36.145 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-04-25T03:20:36.146 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-04-25T03:20:36.146 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /dev 2024-04-25T03:20:36.146 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:20:36.566 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 81/109 2024-04-25T03:20:38.364 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 81/109 2024-04-25T03:20:38.567 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 82/109 2024-04-25T03:20:38.710 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 78/109 2024-04-25T03:20:38.855 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 82/109 2024-04-25T03:20:38.941 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 78/109 2024-04-25T03:20:39.082 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 83/109 2024-04-25T03:20:39.629 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 83/109 2024-04-25T03:20:39.671 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 78/109 2024-04-25T03:20:39.824 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 84/109 2024-04-25T03:20:39.872 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 84/109 2024-04-25T03:20:39.983 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 79/109 2024-04-25T03:20:40.115 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 85/109 2024-04-25T03:20:40.183 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 79/109 2024-04-25T03:20:40.337 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 86/109 2024-04-25T03:20:40.373 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 80/109 2024-04-25T03:20:40.523 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 87/109 2024-04-25T03:20:40.620 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 87/109 2024-04-25T03:20:40.824 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-8.el8.x86_64 88/109 2024-04-25T03:20:41.459 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 89/109 2024-04-25T03:20:41.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 89/109 2024-04-25T03:20:41.752 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2424.gb98aab6c.e 90/109 2024-04-25T03:20:41.950 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2424.gb98aab6c. 91/109 2024-04-25T03:20:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noa 92/109 2024-04-25T03:20:43.328 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noa 92/109 2024-04-25T03:20:43.583 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.n 93/109 2024-04-25T03:20:44.667 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.n 93/109 2024-04-25T03:20:44.847 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98 94/109 2024-04-25T03:20:44.975 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98 94/109 2024-04-25T03:20:45.365 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2424.gb98aab6c.el 95/109 2024-04-25T03:20:45.583 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 96/109 2024-04-25T03:20:45.649 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 96/109 2024-04-25T03:20:45.822 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 97/109 2024-04-25T03:20:45.850 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 97/109 2024-04-25T03:20:51.812 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-2424.gb98aab6c.el8.x86_64 98/109 2024-04-25T03:20:53.544 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-2424.gb98aab6c.el8.x86_64 99/109 2024-04-25T03:20:53.732 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 100/109 2024-04-25T03:20:53.793 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 100/109 2024-04-25T03:20:53.955 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2424.gb98aa 101/109 2024-04-25T03:20:54.376 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2424.gb98aa 101/109 2024-04-25T03:20:54.555 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 102/109 2024-04-25T03:20:54.686 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 102/109 2024-04-25T03:20:54.869 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 103/109 2024-04-25T03:20:55.072 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-2424.gb98aab6c.el8.x86_ 104/109 2024-04-25T03:20:55.255 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 105/109 2024-04-25T03:20:55.439 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 106/109 2024-04-25T03:20:55.616 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : pv-1.6.6-7.el8.x86_64 107/109 2024-04-25T03:20:55.616 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 108/109 2024-04-25T03:20:55.860 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 108/109 2024-04-25T03:20:55.861 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 109/109 2024-04-25T03:20:56.314 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 109/109 2024-04-25T03:20:56.314 INFO:teuthology.orchestra.run.smithi173.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-25T03:20:56.314 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:20:56.950 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-2424.gb98aab6c.el8.x86_64 1/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 2/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 3/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 4/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2424.gb98aa 5/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 6/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 7/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 8/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 9/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 10/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 11/109 2024-04-25T03:20:56.951 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-2424.gb98aab6c.el8.x86_64 12/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-2424.gb98aab6c.el8.x86_ 13/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 14/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 15/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-2424.gb98aab6c.el8.x86_6 16/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86 17/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 18/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2424.gb98aab6c.el 19/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-2424.gb98aab6c.el8. 20/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-2424.gb98aab6c.el8.x86_6 21/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-2424.gb98aab6c.el8.x86_64 22/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 23/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 24/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 25/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 26/109 2024-04-25T03:20:56.952 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 27/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2424.gb98aab6c. 28/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noa 29/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.n 30/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98 31/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2424.gb98aab6c.el 32/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 33/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2424.gb98aab6c.e 34/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 35/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 36/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-8.el8.x86_64 37/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 38/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 39/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 40/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 41/109 2024-04-25T03:20:56.953 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 42/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 43/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 44/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 45/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 46/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 47/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 48/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 49/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 50/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 51/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 52/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 53/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 54/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 55/109 2024-04-25T03:20:56.954 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 56/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 57/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 58/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 59/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 60/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 61/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 62/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 63/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 64/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 65/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 66/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 67/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 68/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 69/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 70/109 2024-04-25T03:20:56.955 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 71/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 72/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 73/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 74/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 75/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : pv-1.6.6-7.el8.x86_64 76/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 77/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 78/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 79/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 80/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 81/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 82/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 83/109 2024-04-25T03:20:56.956 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 84/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 85/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 86/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 87/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 88/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 89/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 90/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 91/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-2.6-1.el8.noarch 92/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 93/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 94/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 95/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 96/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 97/109 2024-04-25T03:20:56.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 98/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 99/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 100/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 101/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 102/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 103/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 104/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 105/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 106/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 107/109 2024-04-25T03:20:56.958 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 108/109 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 109/109 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.499 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.500 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.7-19.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-8.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-25T03:20:58.501 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.5.0-1.el8.noarch 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: pv-1.6.6-7.el8.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-25T03:20:58.502 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-25T03:20:58.503 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-2.6-1.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-25T03:20:58.504 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-25T03:20:58.505 INFO:teuthology.orchestra.run.smithi173.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:20:58.506 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-25T03:20:58.507 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-25T03:20:58.507 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-25T03:20:58.507 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:20:58.507 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-25T03:20:58.910 DEBUG:teuthology.parallel:result is None 2024-04-25T03:21:01.581 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 80/109 2024-04-25T03:21:01.581 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-04-25T03:21:01.581 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /run 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /dev 2024-04-25T03:21:01.582 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:01.960 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 81/109 2024-04-25T03:21:03.770 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 81/109 2024-04-25T03:21:04.002 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 82/109 2024-04-25T03:21:04.291 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 82/109 2024-04-25T03:21:04.476 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 83/109 2024-04-25T03:21:05.033 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 83/109 2024-04-25T03:21:05.226 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 84/109 2024-04-25T03:21:05.273 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 84/109 2024-04-25T03:21:05.493 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 85/109 2024-04-25T03:21:05.714 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 86/109 2024-04-25T03:21:05.900 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 87/109 2024-04-25T03:21:05.994 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 87/109 2024-04-25T03:21:06.189 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-8.el8.x86_64 88/109 2024-04-25T03:21:06.871 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 89/109 2024-04-25T03:21:07.038 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 89/109 2024-04-25T03:21:07.161 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2424.gb98aab6c.e 90/109 2024-04-25T03:21:07.375 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2424.gb98aab6c. 91/109 2024-04-25T03:21:07.608 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noa 92/109 2024-04-25T03:21:08.787 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noa 92/109 2024-04-25T03:21:09.025 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.n 93/109 2024-04-25T03:21:10.112 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.n 93/109 2024-04-25T03:21:10.265 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98 94/109 2024-04-25T03:21:10.393 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98 94/109 2024-04-25T03:21:10.827 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2424.gb98aab6c.el 95/109 2024-04-25T03:21:11.034 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 96/109 2024-04-25T03:21:11.100 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 96/109 2024-04-25T03:21:11.274 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 97/109 2024-04-25T03:21:11.300 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 97/109 2024-04-25T03:21:17.290 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:19.0.0-2424.gb98aab6c.el8.x86_64 98/109 2024-04-25T03:21:19.022 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:19.0.0-2424.gb98aab6c.el8.x86_64 99/109 2024-04-25T03:21:19.223 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 100/109 2024-04-25T03:21:19.284 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 100/109 2024-04-25T03:21:19.472 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2424.gb98aa 101/109 2024-04-25T03:21:19.896 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2424.gb98aa 101/109 2024-04-25T03:21:20.081 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 102/109 2024-04-25T03:21:20.213 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 102/109 2024-04-25T03:21:20.428 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 103/109 2024-04-25T03:21:20.614 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:19.0.0-2424.gb98aab6c.el8.x86_ 104/109 2024-04-25T03:21:20.814 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 105/109 2024-04-25T03:21:21.015 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 106/109 2024-04-25T03:21:21.183 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : pv-1.6.6-7.el8.x86_64 107/109 2024-04-25T03:21:21.184 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 108/109 2024-04-25T03:21:21.445 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 108/109 2024-04-25T03:21:21.445 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 109/109 2024-04-25T03:21:22.005 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 109/109 2024-04-25T03:21:22.005 INFO:teuthology.orchestra.run.smithi188.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-25T03:21:22.005 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:22.670 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:19.0.0-2424.gb98aab6c.el8.x86_64 1/109 2024-04-25T03:21:22.670 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 2/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 3/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 4/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2424.gb98aa 5/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 6/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 7/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 8/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 9/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 10/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 11/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:19.0.0-2424.gb98aab6c.el8.x86_64 12/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:19.0.0-2424.gb98aab6c.el8.x86_ 13/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 14/109 2024-04-25T03:21:22.671 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 15/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:19.0.0-2424.gb98aab6c.el8.x86_6 16/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86 17/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 18/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2424.gb98aab6c.el 19/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:19.0.0-2424.gb98aab6c.el8. 20/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:19.0.0-2424.gb98aab6c.el8.x86_6 21/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:19.0.0-2424.gb98aab6c.el8.x86_64 22/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 23/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 24/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 25/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 26/109 2024-04-25T03:21:22.672 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 27/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2424.gb98aab6c. 28/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noa 29/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.n 30/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98 31/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2424.gb98aab6c.el 32/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 33/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2424.gb98aab6c.e 34/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 35/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 36/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-8.el8.x86_64 37/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 38/109 2024-04-25T03:21:22.681 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 39/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 40/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 41/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 42/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 43/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 44/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 45/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 46/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 47/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 48/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 49/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 50/109 2024-04-25T03:21:22.682 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 51/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 52/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 53/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 54/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 55/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x 56/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 57/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 58/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 59/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 60/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 61/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 62/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 63/109 2024-04-25T03:21:22.683 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 64/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 65/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 66/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 67/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 68/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 69/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.3.4-12.el8.x86_64 70/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 71/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 72/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 73/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 74/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.5.0-1.el8.noarch 75/109 2024-04-25T03:21:22.684 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : pv-1.6.6-7.el8.x86_64 76/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 77/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 78/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 79/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 80/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 81/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 82/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 83/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 84/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 85/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 86/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 87/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 88/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 89/109 2024-04-25T03:21:22.685 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 90/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 91/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-2.6-1.el8.noarch 92/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 93/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 94/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 95/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 96/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 97/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 98/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 99/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 100/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 101/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 102/109 2024-04-25T03:21:22.686 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 103/109 2024-04-25T03:21:22.687 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 104/109 2024-04-25T03:21:22.687 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 105/109 2024-04-25T03:21:22.687 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 106/109 2024-04-25T03:21:22.687 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 107/109 2024-04-25T03:21:22.687 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 108/109 2024-04-25T03:21:24.184 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 109/109 2024-04-25T03:21:24.184 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:24.184 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-04-25T03:21:24.184 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.184 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.185 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:19.0.0-2424.gb98aab6c.el8.noarch 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: fuse-2.9.7-19.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-8.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.186 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.3.4-12.el8.x86_64 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.5.0-1.el8.noarch 2024-04-25T03:21:24.187 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.8.2-3.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.15-4.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: pv-1.6.6-7.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.5.1-7.el8.noarch 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-25T03:21:24.188 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.10.1-4.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-25T03:21:24.189 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-packaging-16.8-9.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-2.6-1.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-25T03:21:24.190 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.4.1-12.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-25T03:21:24.191 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:19.0.0-2424.gb98aab6c.el8.x86_64 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-1.el8.x86_64 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.13.0-2.el8.x86_64 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-04-25T03:21:24.192 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:24.193 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-04-25T03:21:24.610 DEBUG:teuthology.parallel:result is None 2024-04-25T03:21:24.611 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-25T03:21:24.753 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T03:21:24.814 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-2424.gb98aab6c.el8 2024-04-25T03:21:24.815 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2424.gb98aab6c.el8 2024-04-25T03:21:24.815 INFO:teuthology.task.install:The correct ceph version 19.0.0-2424.gb98aab6c is installed. 2024-04-25T03:21:24.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b98aab6c00f483888c049d0d8e7bc507e115dd86 2024-04-25T03:21:25.060 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T03:21:25.117 INFO:teuthology.orchestra.run.smithi188.stdout:19.0.0-2424.gb98aab6c.el8 2024-04-25T03:21:25.117 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2424.gb98aab6c.el8 2024-04-25T03:21:25.117 INFO:teuthology.task.install:The correct ceph version 19.0.0-2424.gb98aab6c is installed. 2024-04-25T03:21:25.119 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T03:21:25.120 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:25.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T03:21:25.197 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:25.198 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T03:21:25.267 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T03:21:25.267 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:25.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T03:21:25.338 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T03:21:25.446 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:25.446 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T03:21:25.512 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T03:21:25.618 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T03:21:25.619 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:25.619 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T03:21:25.685 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T03:21:25.791 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:25.791 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T03:21:25.856 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T03:21:25.963 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T03:21:25.963 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:25.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T03:21:26.030 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T03:21:26.135 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:26.135 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T03:21:26.201 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T03:21:26.308 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T03:21:26.390 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T03:21:26.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2024-04-25T03:21:26.394 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 777 /var/log/ceph 2024-04-25T03:21:26.460 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T03:21:26.460 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T03:21:26.464 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T03:21:26.529 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T03:21:26.530 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T03:21:26.533 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T03:21:26.610 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-04-25T03:21:26.610 INFO:tasks.ceph:config {'conf': {'client': {'rbd default map options': 'ms_mode=crc,rxbounce'}, 'global': {'mon warn on pool no app': False, 'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T03:21:26.610 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443', 'branch': 'squid', 'description': 'krbd/mirror/{bluestore-bitmap clusters/2-node conf install/ceph ms_mode$/{crc-rxbounce} tasks/compare-mirror-images}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672443', 'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': '2 ceph clusters with 1 mon, 1 mgr and 3 osd each'}], 'name': 'teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'rbd default map options': 'ms_mode=crc,rxbounce'}, 'global': {'mon warn on pool no app': False, 'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2'], ['cluster2.mon.a', 'cluster2.mgr.x', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster2.client.mirror', 'cluster2.client.mirror.0']], 'seed': 3056, 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'sleep_before_teardown': 0, 'subset': '0/4', 'suite': 'krbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'targets': {'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEE4UpcTt0SOnbcfyMKwV4DsXIWvHxfC6eU1pVdzrNrg6NXt4UllBl9EMkpqFCKfJ7+7WOztKYtA8MjaHOAAGC0=', 'smithi188.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKLEC0jIcC1bw/OnyBAENWc1L9joCrlM1lzqES0/EyFKQf9xlK/0cgUcDhqfNW8nF/7ttwllaF53FrMpKlNFiCg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'rbd default map options': 'ms_mode=crc,rxbounce'}, 'global': {'mon warn on pool no app': False, 'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster2'}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': False}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': False}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'krbd', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:48:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T03:21:26.611 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-25T03:21:26.615 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-04-25T03:21:26.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T03:21:26.700 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T03:21:26.766 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:26.766 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T03:21:26.809 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T03:21:26.810 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-04-25T03:21:26.894 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 444 Links: 1 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 03:20:56.515783793 +0000 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 03:17:36.660141998 +0000 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 03:17:36.660141998 +0000 2024-04-25T03:21:26.895 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 03:17:36.637142496 +0000 2024-04-25T03:21:26.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T03:21:26.999 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T03:21:27.000 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T03:21:27.000 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000466858 s, 1.1 MB/s 2024-04-25T03:21:27.002 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T03:21:27.048 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-04-25T03:21:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T03:21:27.133 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:21:27.134 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 450 Links: 1 2024-04-25T03:21:27.134 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:21:27.134 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:21:27.134 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 03:20:56.515783793 +0000 2024-04-25T03:21:27.134 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 03:17:37.182130691 +0000 2024-04-25T03:21:27.134 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 03:17:37.182130691 +0000 2024-04-25T03:21:27.134 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 03:17:37.155131275 +0000 2024-04-25T03:21:27.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T03:21:27.237 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T03:21:27.237 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T03:21:27.237 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000368161 s, 1.4 MB/s 2024-04-25T03:21:27.239 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T03:21:27.285 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-04-25T03:21:27.329 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 456 Links: 1 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 03:20:56.515783793 +0000 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 03:17:37.677119968 +0000 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 03:17:37.677119968 +0000 2024-04-25T03:21:27.330 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 03:17:37.651120531 +0000 2024-04-25T03:21:27.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T03:21:27.432 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T03:21:27.432 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T03:21:27.432 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000287492 s, 1.8 MB/s 2024-04-25T03:21:27.435 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T03:21:27.480 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-04-25T03:21:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T03:21:27.524 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:21:27.524 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 6h/6d Inode: 462 Links: 1 2024-04-25T03:21:27.525 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:21:27.525 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:21:27.525 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 03:20:56.515783793 +0000 2024-04-25T03:21:27.525 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 03:17:38.194108769 +0000 2024-04-25T03:21:27.525 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 03:17:38.194108769 +0000 2024-04-25T03:21:27.525 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 03:17:38.173109224 +0000 2024-04-25T03:21:27.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T03:21:27.628 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T03:21:27.628 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T03:21:27.628 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000304639 s, 1.7 MB/s 2024-04-25T03:21:27.630 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T03:21:27.676 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:21:27.676 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-25T03:21:27.676 INFO:tasks.ceph:Generating config... 2024-04-25T03:21:27.677 INFO:tasks.ceph:[client] rbd default map options = ms_mode=crc,rxbounce 2024-04-25T03:21:27.677 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-25T03:21:27.678 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-25T03:21:27.678 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T03:21:27.678 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T03:21:27.678 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T03:21:27.678 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T03:21:27.678 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-25T03:21:27.678 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T03:21:27.679 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T03:21:27.679 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-25T03:21:27.679 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T03:21:27.679 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T03:21:27.679 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T03:21:27.679 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T03:21:27.679 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T03:21:27.679 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-04-25T03:21:27.679 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-04-25T03:21:27.818 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/cluster1.keyring 2024-04-25T03:21:27.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-04-25T03:21:27.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-25T03:21:27.988 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.173')] 2024-04-25T03:21:27.988 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.173', 'mon warn on pool no app': False, 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default map options': 'ms_mode=crc,rxbounce'}, 'mon.a': {}} 2024-04-25T03:21:27.989 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:27.989 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T03:21:28.032 DEBUG:teuthology.orchestra.run.smithi173:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.173 --print /home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:28.155 INFO:teuthology.orchestra.run.smithi173.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:28.155 INFO:teuthology.orchestra.run.smithi173.stdout:monmaptool: generated fsid 2d0af0d3-7b88-4ef9-a498-864f18964d8a 2024-04-25T03:21:28.155 INFO:teuthology.orchestra.run.smithi173.stdout:setting min_mon_release = quincy 2024-04-25T03:21:28.155 INFO:teuthology.orchestra.run.smithi173.stdout:epoch 0 2024-04-25T03:21:28.155 INFO:teuthology.orchestra.run.smithi173.stdout:fsid 2d0af0d3-7b88-4ef9-a498-864f18964d8a 2024-04-25T03:21:28.155 INFO:teuthology.orchestra.run.smithi173.stdout:last_changed 2024-04-25T03:21:28.154072+0000 2024-04-25T03:21:28.156 INFO:teuthology.orchestra.run.smithi173.stdout:created 2024-04-25T03:21:28.154072+0000 2024-04-25T03:21:28.156 INFO:teuthology.orchestra.run.smithi173.stdout:min_mon_release 17 (quincy) 2024-04-25T03:21:28.156 INFO:teuthology.orchestra.run.smithi173.stdout:election_strategy: 1 2024-04-25T03:21:28.156 INFO:teuthology.orchestra.run.smithi173.stdout:0: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.a 2024-04-25T03:21:28.156 INFO:teuthology.orchestra.run.smithi173.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-04-25T03:21:28.158 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T03:21:28.202 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 2d0af0d3-7b88-4ef9-a498-864f18964d8a... 2024-04-25T03:21:28.203 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-25T03:21:28.247 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-04-25T03:21:28.354 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.355 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T03:21:28.356 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = 172.21.15.173 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on pool no app = False 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on skipped message = False 2024-04-25T03:21:28.357 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 2d0af0d3-7b88-4ef9-a498-864f18964d8a 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug trim objects = true 2024-04-25T03:21:28.358 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: bdev async discard = True 2024-04-25T03:21:28.359 INFO:teuthology.orchestra.run.smithi173.stdout: bdev enable discard = True 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore allocator = bitmap 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 1/20 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 1/20 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 4/10 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2024-04-25T03:21:28.360 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:21:28.361 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2024-04-25T03:21:28.362 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout: rbd default map options = "ms_mode=crc,rxbounce" 2024-04-25T03:21:28.363 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout:[global] 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: chdir = "" 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: auth supported = cephx 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: filestore xattr use omap = true 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: mon clock drift allowed = 1.000 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: osd crush chooseleaf type = 0 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: auth debug = true 2024-04-25T03:21:28.367 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on old message = true 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on bug = true 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: mon pg warn max object skew = 0 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default size = 2 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow primary affinity = true 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow pg remap = true 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on legacy crush tunables = false 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on crush straw calc version zero = false 2024-04-25T03:21:28.368 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on no sortbitwise = false 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on osd down out interval zero = false 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on too few osds = false 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon_allow_pool_size_one = true 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: osd default data pool replay window = 5 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon allow pool delete = true 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon cluster log file level = debug 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: debug asserts on shutdown = true 2024-04-25T03:21:28.369 INFO:teuthology.orchestra.run.smithi188.stdout: mon health detail to clog = false 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: mon host = 172.21.15.173 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on pool no app = False 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on skipped message = False 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: fsid = 2d0af0d3-7b88-4ef9-a498-864f18964d8a 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout:[osd] 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: osd journal size = 100 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub load threshold = 5.0 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub max interval = 600 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock profile = high_recovery_ops 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: osd recover clone overlap = true 2024-04-25T03:21:28.370 INFO:teuthology.orchestra.run.smithi188.stdout: osd recovery max chunk = 1048576 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug shutdown = true 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug op order = true 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify stray on activate = true 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug trim objects = true 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd open classes on start = true 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug pg log writeout = true 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd deep scrub update digest min age = 30 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: osd map max advance = 10 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: journal zero on create = true 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: filestore ondisk finisher threads = 3 2024-04-25T03:21:28.371 INFO:teuthology.orchestra.run.smithi188.stdout: filestore apply finisher threads = 3 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: bdev debug aio = true 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug misdirected ops = true 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: bdev async discard = True 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: bdev enable discard = True 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore allocator = bitmap 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore block size = 96636764160 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore fsck on mount = True 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluefs = 1/20 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluestore = 1/20 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: debug osd = 20 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: debug rocksdb = 4/10 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T03:21:28.372 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd full ratio = 0.9 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd nearfull ratio = 0.8 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: osd failsafe full ratio = 0.95 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: osd objectstore = bluestore 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr] 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: debug mgr = 20 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: mgr/telemetry/nag = false 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout:[mon] 2024-04-25T03:21:28.373 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: debug paxos = 20 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: mon data avail warn = 5 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: mon mgr mkfs grace = 240 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd reporter subtree level = osd 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd prime pg temp = true 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.374 INFO:teuthology.orchestra.run.smithi188.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: # 1m isn't quite enough 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: mon_down_mkfs_grace = 2m 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_filestore_osds = false 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout:[client] 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: rgw cache enabled = true 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable ops log = true 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable usage log = true 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T03:21:28.375 INFO:teuthology.orchestra.run.smithi188.stdout: rbd default map options = "ms_mode=crc,rxbounce" 2024-04-25T03:21:28.376 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.a] 2024-04-25T03:21:28.394 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-04-25T03:21:28.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-04-25T03:21:28.504 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T03:21:28.504 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:28.505 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-04-25T03:21:28.548 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:28.548 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-04-25T03:21:28.633 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T03:21:28.633 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:28.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-25T03:21:28.633 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-25T03:21:28.762 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:28.763 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:28.847 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T03:21:28.847 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:28.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/cluster1.keyring 2024-04-25T03:21:28.847 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-04-25T03:21:28.937 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:28.937 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:29.022 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T03:21:29.022 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T03:21:29.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-25T03:21:29.140 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-04-25T03:21:29.146 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T03:21:29.146 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T03:21:29.146 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-04-25T03:21:29.281 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-04-25T03:21:29.312 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-25T03:21:29.406 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-04-25T03:21:29.435 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T03:21:29.436 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-25T03:21:29.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-04-25T03:21:29.503 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:21:29.503 INFO:tasks.ceph:role: cluster1.osd.0 2024-04-25T03:21:29.503 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T03:21:29.503 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T03:21:29.576 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T03:21:29.948 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-04-25T03:21:29.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-04-25T03:21:29.951 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-04-25T03:21:30.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-04-25T03:21:30.197 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-04-25T03:21:30.263 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:21:30.264 INFO:tasks.ceph:role: cluster1.osd.1 2024-04-25T03:21:30.264 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T03:21:30.264 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T03:21:30.337 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T03:21:30.337 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T03:21:30.338 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T03:21:30.688 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-04-25T03:21:30.692 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-04-25T03:21:30.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-04-25T03:21:30.808 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-04-25T03:21:30.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-04-25T03:21:30.946 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:21:30.946 INFO:tasks.ceph:role: cluster1.osd.2 2024-04-25T03:21:30.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T03:21:30.947 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T03:21:31.020 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T03:21:31.020 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T03:21:31.021 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T03:21:31.398 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-04-25T03:21:31.402 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-04-25T03:21:31.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-04-25T03:21:31.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-04-25T03:21:31.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:31.716 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:31.714+0000 7f2c53eab7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-04-25T03:21:31.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:31.715+0000 7f2c53eab7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-04-25T03:21:31.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:31.715+0000 7f2c53eab7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-25T03:21:31.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:31.715+0000 7f2c53eab7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-25T03:21:31.717 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:31.715+0000 7f2c53eab7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-04-25T03:21:31.727 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:31.724+0000 7f2c53eab7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-04-25T03:21:34.663 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-04-25T03:21:34.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:34.839 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:34.836+0000 7ffaa892c7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-04-25T03:21:34.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:34.837+0000 7ffaa892c7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-04-25T03:21:34.840 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:34.837+0000 7ffaa892c7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-25T03:21:34.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:34.837+0000 7ffaa892c7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-25T03:21:34.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:34.837+0000 7ffaa892c7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-04-25T03:21:34.841 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:34.839+0000 7ffaa892c7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-04-25T03:21:37.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-04-25T03:21:37.871 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:37.970+0000 7f268d2b87c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-04-25T03:21:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:37.970+0000 7f268d2b87c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-04-25T03:21:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:37.970+0000 7f268d2b87c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-25T03:21:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:37.970+0000 7f268d2b87c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-25T03:21:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:37.970+0000 7f268d2b87c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-04-25T03:21:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T03:21:37.972+0000 7f268d2b87c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-04-25T03:21:40.911 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-04-25T03:21:40.978 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T03:21:40.979 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:40.979 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-04-25T03:21:41.045 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:41.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-04-25T03:21:41.151 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:41.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-04-25T03:21:41.259 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:41.259 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-04-25T03:21:41.365 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:41.365 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-04-25T03:21:41.416 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:21:41.416 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-04-25T03:21:41.462 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T03:21:41.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo tee -a /etc/ceph/cluster1.keyring 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.x] 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA5zClmv41ICBAAunWVlIFB0VtxyfiwH5NISA== 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.0] 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA7zClmvS+fKhAAnlXznVeE8qW2S0IvltqiTw== 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.1] 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA+zClm3bv5MRAAbubcGe7hTSF63p8L2AfZpw== 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.2] 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBBzClmbCHyORAAmuvCQ/o7mbzaQVNJxU150A== 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout:[client.mirror] 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA5zClmh2C3EBAAPd/4+sTX6j+dL0nJ2v7pKg== 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout:[client.mirror.0] 2024-04-25T03:21:41.525 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQA5zClmy+EiGBAAkVUpp0mx6QGS9mz2JQgZJg== 2024-04-25T03:21:41.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T03:21:41.672 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T03:21:41.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T03:21:41.884 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T03:21:41.989 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T03:21:42.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T03:21:42.209 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T03:21:42.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-04-25T03:21:42.276 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-04-25T03:21:42.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-04-25T03:21:42.463 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-04-25T03:21:42.507 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-04-25T03:21:42.507 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-04-25T03:21:42.507 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-04-25T03:21:42.551 INFO:tasks.ceph.cluster1.mon.a:Started 2024-04-25T03:21:42.551 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-04-25T03:21:42.551 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-04-25T03:21:42.551 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-04-25T03:21:42.555 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-04-25T03:21:42.555 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T03:21:42.555 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster cluster1 osd crush tunables default 2024-04-25T03:21:42.836 INFO:teuthology.orchestra.run.smithi173.stderr:adjusted tunables profile to default 2024-04-25T03:21:42.852 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T03:21:42.852 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-04-25T03:21:42.853 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:42.853 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-04-25T03:21:42.922 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:42.922 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-04-25T03:21:43.029 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:21:43.029 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-04-25T03:21:43.064 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:43.061+0000 7ff3532d6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T03:21:43.131 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster cluster1 osd new 83fb82c9-be2c-46f4-962d-3d04a08913bf 0 2024-04-25T03:21:43.291 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:43.288+0000 7ff3532d6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T03:21:43.497 INFO:teuthology.orchestra.run.smithi173.stdout:0 2024-04-25T03:21:43.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster cluster1 osd new adefaee8-cbd6-47a4-abdc-55ff3a06d718 1 2024-04-25T03:21:43.735 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:43.732+0000 7ff3532d6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T03:21:43.801 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-04-25T03:21:43.820 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster cluster1 osd new 3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a 2 2024-04-25T03:21:44.104 INFO:teuthology.orchestra.run.smithi173.stdout:2 2024-04-25T03:21:44.122 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-04-25T03:21:44.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-04-25T03:21:44.126 INFO:tasks.ceph.cluster1.osd.0:Started 2024-04-25T03:21:44.126 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-04-25T03:21:44.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-04-25T03:21:44.130 INFO:tasks.ceph.cluster1.osd.1:Started 2024-04-25T03:21:44.130 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-04-25T03:21:44.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-04-25T03:21:44.134 INFO:tasks.ceph.cluster1.osd.2:Started 2024-04-25T03:21:44.134 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-25T03:21:44.437 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:21:44.437 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":5,"fsid":"2d0af0d3-7b88-4ef9-a498-864f18964d8a","created":"2024-04-25T03:21:42.734143+0000","modified":"2024-04-25T03:21:44.102108+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T03:21:44.102108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83fb82c9-be2c-46f4-962d-3d04a08913bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"adefaee8-cbd6-47a4-abdc-55ff3a06d718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:21:44.454 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:44.451+0000 7ff3532d6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T03:21:44.457 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-04-25T03:21:44.457 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T03:21:44.717 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:44.714+0000 7ff3532d6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T03:21:44.760 DEBUG:teuthology.orchestra.run.smithi173:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-25T03:21:44.824 INFO:tasks.ceph.cluster1.osd.1.smithi173.stderr:2024-04-25T03:21:44.821+0000 7f01780b97c0 -1 Falling back to public interface 2024-04-25T03:21:44.824 INFO:tasks.ceph.cluster1.osd.0.smithi173.stderr:2024-04-25T03:21:44.821+0000 7f4e3abf97c0 -1 Falling back to public interface 2024-04-25T03:21:44.862 INFO:tasks.ceph.cluster1.osd.2.smithi173.stderr:2024-04-25T03:21:44.859+0000 7f920a7447c0 -1 Falling back to public interface 2024-04-25T03:21:45.017 INFO:teuthology.misc.health.smithi173.stdout: 2024-04-25T03:21:45.017 INFO:teuthology.misc.health.smithi173.stdout:{"epoch":5,"fsid":"2d0af0d3-7b88-4ef9-a498-864f18964d8a","created":"2024-04-25T03:21:42.734143+0000","modified":"2024-04-25T03:21:44.102108+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T03:21:44.102108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83fb82c9-be2c-46f4-962d-3d04a08913bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"adefaee8-cbd6-47a4-abdc-55ff3a06d718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:21:45.032 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-25T03:21:45.213 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:45.210+0000 7ff3532d6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T03:21:46.047 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.045+0000 7ff3532d6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T03:21:46.141 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.139+0000 7ff3532d6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T03:21:46.313 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.311+0000 7ff3532d6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T03:21:46.528 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.526+0000 7ff3532d6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T03:21:46.626 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.623+0000 7ff3532d6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T03:21:46.726 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.722+0000 7ff3532d6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T03:21:46.825 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.823+0000 7ff3532d6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T03:21:46.940 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:46.937+0000 7ff3532d6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T03:21:47.348 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:47.346+0000 7ff3532d6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T03:21:47.479 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:47.477+0000 7ff3532d6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T03:21:47.700 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:47.698+0000 7ff3532d6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T03:21:47.818 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:47.816+0000 7ff3532d6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T03:21:48.477 INFO:tasks.ceph.cluster1.osd.1.smithi173.stderr:2024-04-25T03:21:48.475+0000 7f01780b97c0 -1 osd.1 0 log_to_monitors true 2024-04-25T03:21:48.498 INFO:tasks.ceph.cluster1.osd.2.smithi173.stderr:2024-04-25T03:21:48.496+0000 7f920a7447c0 -1 osd.2 0 log_to_monitors true 2024-04-25T03:21:48.717 INFO:tasks.ceph.cluster1.osd.0.smithi173.stderr:2024-04-25T03:21:48.715+0000 7f4e3abf97c0 -1 osd.0 0 log_to_monitors true 2024-04-25T03:21:48.728 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:48.726+0000 7ff3532d6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T03:21:48.937 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:48.935+0000 7ff3532d6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T03:21:49.139 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:49.137+0000 7ff3532d6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T03:21:49.679 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:49.676+0000 7ff3532d6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T03:21:49.946 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:49.944+0000 7ff3532d6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T03:21:50.519 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:50.517+0000 7ff3532d6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T03:21:50.727 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:50.725+0000 7ff3532d6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T03:21:51.123 INFO:tasks.ceph.cluster1.osd.1.smithi173.stderr:2024-04-25T03:21:51.121+0000 7f0165400700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T03:21:51.123 INFO:tasks.ceph.cluster1.osd.2.smithi173.stderr:2024-04-25T03:21:51.122+0000 7f91f9800700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T03:21:51.129 INFO:tasks.ceph.cluster1.osd.2.smithi173.stderr:2024-04-25T03:21:51.127+0000 7f91edc00700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T03:21:51.130 INFO:tasks.ceph.cluster1.osd.1.smithi173.stderr:2024-04-25T03:21:51.128+0000 7f015b600700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T03:21:51.336 DEBUG:teuthology.orchestra.run.smithi173:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-25T03:21:51.655 INFO:teuthology.misc.health.smithi173.stdout: 2024-04-25T03:21:51.655 INFO:teuthology.misc.health.smithi173.stdout:{"epoch":8,"fsid":"2d0af0d3-7b88-4ef9-a498-864f18964d8a","created":"2024-04-25T03:21:42.734143+0000","modified":"2024-04-25T03:21:50.534146+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T03:21:44.102108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83fb82c9-be2c-46f4-962d-3d04a08913bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"adefaee8-cbd6-47a4-abdc-55ff3a06d718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:21:51.677 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-25T03:21:51.744 INFO:tasks.ceph.cluster1.mgr.x.smithi173.stderr:2024-04-25T03:21:51.742+0000 7ff309400700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-25T03:21:51.747 INFO:tasks.ceph.cluster1.osd.0.smithi173.stderr:2024-04-25T03:21:51.745+0000 7f4e27e00700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T03:21:51.757 INFO:tasks.ceph.cluster1.osd.0.smithi173.stderr:2024-04-25T03:21:51.755+0000 7f4e1e000700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T03:21:57.981 DEBUG:teuthology.orchestra.run.smithi173:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-04-25T03:21:58.384 INFO:teuthology.misc.health.smithi173.stdout: 2024-04-25T03:21:58.384 INFO:teuthology.misc.health.smithi173.stdout:{"epoch":12,"fsid":"2d0af0d3-7b88-4ef9-a498-864f18964d8a","created":"2024-04-25T03:21:42.734143+0000","modified":"2024-04-25T03:21:55.126697+0000","last_up_change":"2024-04-25T03:21:52.123612+0000","last_in_change":"2024-04-25T03:21:44.102108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:21:52.820161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83fb82c9-be2c-46f4-962d-3d04a08913bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6805","nonce":38674190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6810","nonce":38674190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6815","nonce":38674190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6813","nonce":38674190}]},"public_addr":"172.21.15.173:6805/38674190","cluster_addr":"172.21.15.173:6810/38674190","heartbeat_back_addr":"172.21.15.173:6815/38674190","heartbeat_front_addr":"172.21.15.173:6813/38674190","state":["exists","up"]},{"osd":1,"uuid":"adefaee8-cbd6-47a4-abdc-55ff3a06d718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6801","nonce":2340622026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6804","nonce":2340622026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6811","nonce":2340622026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6808","nonce":2340622026}]},"public_addr":"172.21.15.173:6801/2340622026","cluster_addr":"172.21.15.173:6804/2340622026","heartbeat_back_addr":"172.21.15.173:6811/2340622026","heartbeat_front_addr":"172.21.15.173:6808/2340622026","state":["exists","up"]},{"osd":2,"uuid":"3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6817","nonce":193268527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6819","nonce":193268527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6823","nonce":193268527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6821","nonce":193268527}]},"public_addr":"172.21.15.173:6817/193268527","cluster_addr":"172.21.15.173:6819/193268527","heartbeat_back_addr":"172.21.15.173:6823/193268527","heartbeat_front_addr":"172.21.15.173:6821/193268527","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.716608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.500194+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:51.122866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:21:58.403 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-25T03:21:58.403 INFO:tasks.ceph:Creating RBD pool 2024-04-25T03:21:58.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-04-25T03:21:59.444 INFO:teuthology.orchestra.run.smithi173.stderr:pool 'rbd' created 2024-04-25T03:21:59.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-25T03:22:00.450 INFO:teuthology.orchestra.run.smithi173.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-25T03:22:00.471 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-04-25T03:22:00.471 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-04-25T03:22:00.472 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-04-25T03:22:00.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-04-25T03:22:00.472 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T03:22:00.949 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:00.968 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":1815036534},{"type":"v1","addr":"172.21.15.173:6825","nonce":1815036534}]},"active_addr":"172.21.15.173:6825/1815036534","active_change":"2024-04-25T03:21:50.729087+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":2753897229}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":497886018}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":2034729110}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.173:0","nonce":2712474432}]}]} 2024-04-25T03:22:00.970 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-04-25T03:22:00.970 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-04-25T03:22:00.970 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-25T03:22:01.380 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:01.380 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":14,"fsid":"2d0af0d3-7b88-4ef9-a498-864f18964d8a","created":"2024-04-25T03:21:42.734143+0000","modified":"2024-04-25T03:22:00.438259+0000","last_up_change":"2024-04-25T03:21:52.123612+0000","last_in_change":"2024-04-25T03:21:44.102108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:21:52.820161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T03:21:58.844332+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83fb82c9-be2c-46f4-962d-3d04a08913bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6805","nonce":38674190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6810","nonce":38674190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6815","nonce":38674190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6813","nonce":38674190}]},"public_addr":"172.21.15.173:6805/38674190","cluster_addr":"172.21.15.173:6810/38674190","heartbeat_back_addr":"172.21.15.173:6815/38674190","heartbeat_front_addr":"172.21.15.173:6813/38674190","state":["exists","up"]},{"osd":1,"uuid":"adefaee8-cbd6-47a4-abdc-55ff3a06d718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6801","nonce":2340622026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6804","nonce":2340622026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6811","nonce":2340622026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6808","nonce":2340622026}]},"public_addr":"172.21.15.173:6801/2340622026","cluster_addr":"172.21.15.173:6804/2340622026","heartbeat_back_addr":"172.21.15.173:6811/2340622026","heartbeat_front_addr":"172.21.15.173:6808/2340622026","state":["exists","up"]},{"osd":2,"uuid":"3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6817","nonce":193268527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6819","nonce":193268527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6823","nonce":193268527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6821","nonce":193268527}]},"public_addr":"172.21.15.173:6817/193268527","cluster_addr":"172.21.15.173:6819/193268527","heartbeat_back_addr":"172.21.15.173:6823/193268527","heartbeat_front_addr":"172.21.15.173:6821/193268527","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.716608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.500194+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:51.122866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:01.401 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-04-25T03:22:01.401 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-25T03:22:01.815 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:01.816 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":15,"fsid":"2d0af0d3-7b88-4ef9-a498-864f18964d8a","created":"2024-04-25T03:21:42.734143+0000","modified":"2024-04-25T03:22:01.442594+0000","last_up_change":"2024-04-25T03:21:52.123612+0000","last_in_change":"2024-04-25T03:21:44.102108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:21:52.820161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T03:21:58.844332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83fb82c9-be2c-46f4-962d-3d04a08913bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6805","nonce":38674190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6810","nonce":38674190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6815","nonce":38674190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6813","nonce":38674190}]},"public_addr":"172.21.15.173:6805/38674190","cluster_addr":"172.21.15.173:6810/38674190","heartbeat_back_addr":"172.21.15.173:6815/38674190","heartbeat_front_addr":"172.21.15.173:6813/38674190","state":["exists","up"]},{"osd":1,"uuid":"adefaee8-cbd6-47a4-abdc-55ff3a06d718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6801","nonce":2340622026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6804","nonce":2340622026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6811","nonce":2340622026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6808","nonce":2340622026}]},"public_addr":"172.21.15.173:6801/2340622026","cluster_addr":"172.21.15.173:6804/2340622026","heartbeat_back_addr":"172.21.15.173:6811/2340622026","heartbeat_front_addr":"172.21.15.173:6808/2340622026","state":["exists","up"]},{"osd":2,"uuid":"3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6817","nonce":193268527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6819","nonce":193268527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6823","nonce":193268527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6821","nonce":193268527}]},"public_addr":"172.21.15.173:6817/193268527","cluster_addr":"172.21.15.173:6819/193268527","heartbeat_back_addr":"172.21.15.173:6823/193268527","heartbeat_front_addr":"172.21.15.173:6821/193268527","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.716608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.500194+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:51.122866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:01.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-04-25T03:22:01.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-04-25T03:22:01.837 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-04-25T03:22:02.110 INFO:teuthology.orchestra.run.smithi173.stdout:38654705667 2024-04-25T03:22:02.111 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-25T03:22:02.112 INFO:teuthology.orchestra.run.smithi173.stdout:38654705667 2024-04-25T03:22:02.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-25T03:22:02.113 INFO:teuthology.orchestra.run.smithi173.stdout:38654705668 2024-04-25T03:22:02.114 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-25T03:22:02.599 INFO:teuthology.orchestra.run.smithi173.stdout:38654705666 2024-04-25T03:22:02.600 INFO:teuthology.orchestra.run.smithi173.stdout:38654705666 2024-04-25T03:22:02.611 INFO:teuthology.orchestra.run.smithi173.stdout:38654705667 2024-04-25T03:22:02.631 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705666 for osd.0 2024-04-25T03:22:02.632 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705666 for osd.2 2024-04-25T03:22:02.633 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.1 2024-04-25T03:22:03.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-04-25T03:22:03.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-04-25T03:22:03.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-04-25T03:22:04.082 INFO:teuthology.orchestra.run.smithi173.stdout:38654705668 2024-04-25T03:22:04.088 INFO:teuthology.orchestra.run.smithi173.stdout:38654705668 2024-04-25T03:22:04.105 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705668 for osd.0 2024-04-25T03:22:04.105 DEBUG:teuthology.parallel:result is None 2024-04-25T03:22:04.110 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705667 got 38654705668 for osd.2 2024-04-25T03:22:04.110 DEBUG:teuthology.parallel:result is None 2024-04-25T03:22:04.127 INFO:teuthology.orchestra.run.smithi173.stdout:38654705669 2024-04-25T03:22:04.148 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705669 for osd.1 2024-04-25T03:22:04.148 DEBUG:teuthology.parallel:result is None 2024-04-25T03:22:04.148 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-25T03:22:04.148 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-25T03:22:04.601 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:04.604 INFO:teuthology.orchestra.run.smithi173.stderr:dumped all 2024-04-25T03:22:04.621 INFO:teuthology.orchestra.run.smithi173.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-04-25T03:22:02.746611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":86764,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283028756,"statfs":{"total":289910292480,"available":289821446144,"internally_reserved":0,"allocated":1572864,"data_stored":1288171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":87223631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000735"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447826+0000","last_change":"2024-04-25T03:22:00.454572+0000","last_active":"2024-04-25T03:22:01.447826+0000","last_peered":"2024-04-25T03:22:01.447826+0000","last_clean":"2024-04-25T03:22:01.447826+0000","last_became_active":"2024-04-25T03:22:00.454212+0000","last_became_peered":"2024-04-25T03:22:00.454212+0000","last_unstale":"2024-04-25T03:22:01.447826+0000","last_undegraded":"2024-04-25T03:22:01.447826+0000","last_fullsized":"2024-04-25T03:22:01.447826+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:00.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447735+0000","last_change":"2024-04-25T03:22:00.454554+0000","last_active":"2024-04-25T03:22:01.447735+0000","last_peered":"2024-04-25T03:22:01.447735+0000","last_clean":"2024-04-25T03:22:01.447735+0000","last_became_active":"2024-04-25T03:22:00.454254+0000","last_became_peered":"2024-04-25T03:22:00.454254+0000","last_unstale":"2024-04-25T03:22:01.447735+0000","last_undegraded":"2024-04-25T03:22:01.447735+0000","last_fullsized":"2024-04-25T03:22:01.447735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:08.518908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447884+0000","last_change":"2024-04-25T03:22:00.453301+0000","last_active":"2024-04-25T03:22:01.447884+0000","last_peered":"2024-04-25T03:22:01.447884+0000","last_clean":"2024-04-25T03:22:01.447884+0000","last_became_active":"2024-04-25T03:22:00.453022+0000","last_became_peered":"2024-04-25T03:22:00.453022+0000","last_unstale":"2024-04-25T03:22:01.447884+0000","last_undegraded":"2024-04-25T03:22:01.447884+0000","last_fullsized":"2024-04-25T03:22:01.447884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:38.378533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:02.483620+0000","last_change":"2024-04-25T03:22:00.451023+0000","last_active":"2024-04-25T03:22:02.483620+0000","last_peered":"2024-04-25T03:22:02.483620+0000","last_clean":"2024-04-25T03:22:02.483620+0000","last_became_active":"2024-04-25T03:22:00.450704+0000","last_became_peered":"2024-04-25T03:22:00.450704+0000","last_unstale":"2024-04-25T03:22:02.483620+0000","last_undegraded":"2024-04-25T03:22:02.483620+0000","last_fullsized":"2024-04-25T03:22:02.483620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:05.312724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447539+0000","last_change":"2024-04-25T03:21:54.133861+0000","last_active":"2024-04-25T03:22:01.447539+0000","last_peered":"2024-04-25T03:22:01.447539+0000","last_clean":"2024-04-25T03:22:01.447539+0000","last_became_active":"2024-04-25T03:21:54.133471+0000","last_became_peered":"2024-04-25T03:21:54.133471+0000","last_unstale":"2024-04-25T03:22:01.447539+0000","last_undegraded":"2024-04-25T03:22:01.447539+0000","last_fullsized":"2024-04-25T03:22:01.447539+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_clean_scrub_stamp":"2024-04-25T03:21:53.124515+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:53:23.257337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447747+0000","last_change":"2024-04-25T03:22:00.450555+0000","last_active":"2024-04-25T03:22:01.447747+0000","last_peered":"2024-04-25T03:22:01.447747+0000","last_clean":"2024-04-25T03:22:01.447747+0000","last_became_active":"2024-04-25T03:22:00.450263+0000","last_became_peered":"2024-04-25T03:22:00.450263+0000","last_unstale":"2024-04-25T03:22:01.447747+0000","last_undegraded":"2024-04-25T03:22:01.447747+0000","last_fullsized":"2024-04-25T03:22:01.447747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:21:57.478981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:02.483533+0000","last_change":"2024-04-25T03:22:00.452201+0000","last_active":"2024-04-25T03:22:02.483533+0000","last_peered":"2024-04-25T03:22:02.483533+0000","last_clean":"2024-04-25T03:22:02.483533+0000","last_became_active":"2024-04-25T03:22:00.451642+0000","last_became_peered":"2024-04-25T03:22:00.451642+0000","last_unstale":"2024-04-25T03:22:02.483533+0000","last_undegraded":"2024-04-25T03:22:02.483533+0000","last_fullsized":"2024-04-25T03:22:02.483533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:31.160601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:02.024943+0000","last_change":"2024-04-25T03:22:00.452610+0000","last_active":"2024-04-25T03:22:02.024943+0000","last_peered":"2024-04-25T03:22:02.024943+0000","last_clean":"2024-04-25T03:22:02.024943+0000","last_became_active":"2024-04-25T03:22:00.452291+0000","last_became_peered":"2024-04-25T03:22:00.452291+0000","last_unstale":"2024-04-25T03:22:02.024943+0000","last_undegraded":"2024-04-25T03:22:02.024943+0000","last_fullsized":"2024-04-25T03:22:02.024943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:42:10.308241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447811+0000","last_change":"2024-04-25T03:22:00.453221+0000","last_active":"2024-04-25T03:22:01.447811+0000","last_peered":"2024-04-25T03:22:01.447811+0000","last_clean":"2024-04-25T03:22:01.447811+0000","last_became_active":"2024-04-25T03:22:00.452902+0000","last_became_peered":"2024-04-25T03:22:00.452902+0000","last_unstale":"2024-04-25T03:22:01.447811+0000","last_undegraded":"2024-04-25T03:22:01.447811+0000","last_fullsized":"2024-04-25T03:22:01.447811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:35:05.745342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":724992,"data_stored":627339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":724992,"data_stored":627339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T03:22:04.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-25T03:22:05.028 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:05.031 INFO:teuthology.orchestra.run.smithi173.stderr:dumped all 2024-04-25T03:22:05.048 INFO:teuthology.orchestra.run.smithi173.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-04-25T03:22:04.747008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":86764,"kb_used_data":1536,"kb_used_omap":4,"kb_used_meta":85179,"kb_avail":283028756,"statfs":{"total":289910292480,"available":289821446144,"internally_reserved":0,"allocated":1572864,"data_stored":1288171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":87223631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001132"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447826+0000","last_change":"2024-04-25T03:22:00.454572+0000","last_active":"2024-04-25T03:22:01.447826+0000","last_peered":"2024-04-25T03:22:01.447826+0000","last_clean":"2024-04-25T03:22:01.447826+0000","last_became_active":"2024-04-25T03:22:00.454212+0000","last_became_peered":"2024-04-25T03:22:00.454212+0000","last_unstale":"2024-04-25T03:22:01.447826+0000","last_undegraded":"2024-04-25T03:22:01.447826+0000","last_fullsized":"2024-04-25T03:22:01.447826+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:00.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447735+0000","last_change":"2024-04-25T03:22:00.454554+0000","last_active":"2024-04-25T03:22:01.447735+0000","last_peered":"2024-04-25T03:22:01.447735+0000","last_clean":"2024-04-25T03:22:01.447735+0000","last_became_active":"2024-04-25T03:22:00.454254+0000","last_became_peered":"2024-04-25T03:22:00.454254+0000","last_unstale":"2024-04-25T03:22:01.447735+0000","last_undegraded":"2024-04-25T03:22:01.447735+0000","last_fullsized":"2024-04-25T03:22:01.447735+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:08.518908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447884+0000","last_change":"2024-04-25T03:22:00.453301+0000","last_active":"2024-04-25T03:22:01.447884+0000","last_peered":"2024-04-25T03:22:01.447884+0000","last_clean":"2024-04-25T03:22:01.447884+0000","last_became_active":"2024-04-25T03:22:00.453022+0000","last_became_peered":"2024-04-25T03:22:00.453022+0000","last_unstale":"2024-04-25T03:22:01.447884+0000","last_undegraded":"2024-04-25T03:22:01.447884+0000","last_fullsized":"2024-04-25T03:22:01.447884+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:38.378533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:02.483620+0000","last_change":"2024-04-25T03:22:00.451023+0000","last_active":"2024-04-25T03:22:02.483620+0000","last_peered":"2024-04-25T03:22:02.483620+0000","last_clean":"2024-04-25T03:22:02.483620+0000","last_became_active":"2024-04-25T03:22:00.450704+0000","last_became_peered":"2024-04-25T03:22:00.450704+0000","last_unstale":"2024-04-25T03:22:02.483620+0000","last_undegraded":"2024-04-25T03:22:02.483620+0000","last_fullsized":"2024-04-25T03:22:02.483620+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:05.312724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":145,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447539+0000","last_change":"2024-04-25T03:21:54.133861+0000","last_active":"2024-04-25T03:22:01.447539+0000","last_peered":"2024-04-25T03:22:01.447539+0000","last_clean":"2024-04-25T03:22:01.447539+0000","last_became_active":"2024-04-25T03:21:54.133471+0000","last_became_peered":"2024-04-25T03:21:54.133471+0000","last_unstale":"2024-04-25T03:22:01.447539+0000","last_undegraded":"2024-04-25T03:22:01.447539+0000","last_fullsized":"2024-04-25T03:22:01.447539+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_clean_scrub_stamp":"2024-04-25T03:21:53.124515+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:53:23.257337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447747+0000","last_change":"2024-04-25T03:22:00.450555+0000","last_active":"2024-04-25T03:22:01.447747+0000","last_peered":"2024-04-25T03:22:01.447747+0000","last_clean":"2024-04-25T03:22:01.447747+0000","last_became_active":"2024-04-25T03:22:00.450263+0000","last_became_peered":"2024-04-25T03:22:00.450263+0000","last_unstale":"2024-04-25T03:22:01.447747+0000","last_undegraded":"2024-04-25T03:22:01.447747+0000","last_fullsized":"2024-04-25T03:22:01.447747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:21:57.478981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:02.483533+0000","last_change":"2024-04-25T03:22:00.452201+0000","last_active":"2024-04-25T03:22:02.483533+0000","last_peered":"2024-04-25T03:22:02.483533+0000","last_clean":"2024-04-25T03:22:02.483533+0000","last_became_active":"2024-04-25T03:22:00.451642+0000","last_became_peered":"2024-04-25T03:22:00.451642+0000","last_unstale":"2024-04-25T03:22:02.483533+0000","last_undegraded":"2024-04-25T03:22:02.483533+0000","last_fullsized":"2024-04-25T03:22:02.483533+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:31.160601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:02.024943+0000","last_change":"2024-04-25T03:22:00.452610+0000","last_active":"2024-04-25T03:22:02.024943+0000","last_peered":"2024-04-25T03:22:02.024943+0000","last_clean":"2024-04-25T03:22:02.024943+0000","last_became_active":"2024-04-25T03:22:00.452291+0000","last_became_peered":"2024-04-25T03:22:00.452291+0000","last_unstale":"2024-04-25T03:22:02.024943+0000","last_undegraded":"2024-04-25T03:22:02.024943+0000","last_fullsized":"2024-04-25T03:22:02.024943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:42:10.308241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-25T03:22:01.447811+0000","last_change":"2024-04-25T03:22:00.453221+0000","last_active":"2024-04-25T03:22:01.447811+0000","last_peered":"2024-04-25T03:22:01.447811+0000","last_clean":"2024-04-25T03:22:01.447811+0000","last_became_active":"2024-04-25T03:22:00.452902+0000","last_became_peered":"2024-04-25T03:22:00.452902+0000","last_unstale":"2024-04-25T03:22:01.447811+0000","last_undegraded":"2024-04-25T03:22:01.447811+0000","last_fullsized":"2024-04-25T03:22:01.447811+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:35:05.745342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":724992,"data_stored":627339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":724992,"data_stored":627339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T03:22:05.049 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-25T03:22:05.049 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-04-25T03:22:05.049 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-04-25T03:22:05.049 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-04-25T03:22:05.507 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:05.507 INFO:teuthology.orchestra.run.smithi173.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T03:22:05.528 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-04-25T03:22:05.528 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T03:22:05.539 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-04-25T03:22:05.539 INFO:tasks.ceph:config {'conf': {'client': {'rbd default map options': 'ms_mode=crc,rxbounce'}, 'global': {'mon warn on pool no app': False, 'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T03:22:05.539 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443', 'branch': 'squid', 'description': 'krbd/mirror/{bluestore-bitmap clusters/2-node conf install/ceph ms_mode$/{crc-rxbounce} tasks/compare-mirror-images}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672443', 'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': '2 ceph clusters with 1 mon, 1 mgr and 3 osd each'}], 'name': 'teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'rbd default map options': 'ms_mode=crc,rxbounce'}, 'global': {'mon warn on pool no app': False, 'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2'], ['cluster2.mon.a', 'cluster2.mgr.x', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster2.client.mirror', 'cluster2.client.mirror.0']], 'seed': 3056, 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86', 'sleep_before_teardown': 0, 'subset': '0/4', 'suite': 'krbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'targets': {'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEE4UpcTt0SOnbcfyMKwV4DsXIWvHxfC6eU1pVdzrNrg6NXt4UllBl9EMkpqFCKfJ7+7WOztKYtA8MjaHOAAGC0=', 'smithi188.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKLEC0jIcC1bw/OnyBAENWc1L9joCrlM1lzqES0/EyFKQf9xlK/0cgUcDhqfNW8nF/7ttwllaF53FrMpKlNFiCg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': 1, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'extra_system_packages': ['pv'], 'flavor': 'default', 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'rbd default map options': 'ms_mode=crc,rxbounce'}, 'global': {'mon warn on pool no app': False, 'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'rbd default map options': 'ms_mode=crc,rxbounce'}, 'global': {'mon warn on pool no app': False, 'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b98aab6c00f483888c049d0d8e7bc507e115dd86'}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': False}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': False}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'krbd', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:48:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T03:22:05.539 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-25T03:22:05.544 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-04-25T03:22:05.593 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T03:22:05.629 DEBUG:teuthology.orchestra.run.smithi188:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T03:22:05.700 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:05.700 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T03:22:05.744 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T03:22:05.744 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_1 2024-04-25T03:22:05.829 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T03:22:05.829 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:22:05.829 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 444 Links: 1 2024-04-25T03:22:05.829 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:22:05.829 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:22:05.829 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-25 03:21:22.221637785 +0000 2024-04-25T03:22:05.829 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-25 03:17:36.655364857 +0000 2024-04-25T03:22:05.830 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-25 03:17:36.655364857 +0000 2024-04-25T03:22:05.830 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-25 03:17:36.633365252 +0000 2024-04-25T03:22:05.830 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T03:22:05.933 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-25T03:22:05.933 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-25T03:22:05.934 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.00029903 s, 1.7 MB/s 2024-04-25T03:22:05.936 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T03:22:05.982 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_2 2024-04-25T03:22:06.067 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T03:22:06.067 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:22:06.068 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 450 Links: 1 2024-04-25T03:22:06.068 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:22:06.068 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:22:06.068 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-25 03:21:22.221637785 +0000 2024-04-25T03:22:06.068 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-25 03:17:37.161355769 +0000 2024-04-25T03:22:06.068 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-25 03:17:37.161355769 +0000 2024-04-25T03:22:06.068 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-25 03:17:37.138356183 +0000 2024-04-25T03:22:06.068 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T03:22:06.171 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-25T03:22:06.171 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-25T03:22:06.171 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000269857 s, 1.9 MB/s 2024-04-25T03:22:06.173 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T03:22:06.220 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_3 2024-04-25T03:22:06.264 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T03:22:06.264 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:22:06.264 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 456 Links: 1 2024-04-25T03:22:06.264 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:22:06.264 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:22:06.264 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-25 03:21:22.221637785 +0000 2024-04-25T03:22:06.264 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-25 03:17:37.668346664 +0000 2024-04-25T03:22:06.265 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-25 03:17:37.668346664 +0000 2024-04-25T03:22:06.265 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-25 03:17:37.644347095 +0000 2024-04-25T03:22:06.265 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T03:22:06.367 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-25T03:22:06.367 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-25T03:22:06.367 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000297084 s, 1.7 MB/s 2024-04-25T03:22:06.370 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T03:22:06.415 DEBUG:teuthology.orchestra.run.smithi188:> stat /dev/vg_nvme/lv_4 2024-04-25T03:22:06.502 INFO:teuthology.orchestra.run.smithi188.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T03:22:06.502 INFO:teuthology.orchestra.run.smithi188.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T03:22:06.502 INFO:teuthology.orchestra.run.smithi188.stdout:Device: 6h/6d Inode: 462 Links: 1 2024-04-25T03:22:06.502 INFO:teuthology.orchestra.run.smithi188.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T03:22:06.502 INFO:teuthology.orchestra.run.smithi188.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T03:22:06.503 INFO:teuthology.orchestra.run.smithi188.stdout:Access: 2024-04-25 03:21:22.222637767 +0000 2024-04-25T03:22:06.503 INFO:teuthology.orchestra.run.smithi188.stdout:Modify: 2024-04-25 03:17:38.191337270 +0000 2024-04-25T03:22:06.503 INFO:teuthology.orchestra.run.smithi188.stdout:Change: 2024-04-25 03:17:38.191337270 +0000 2024-04-25T03:22:06.503 INFO:teuthology.orchestra.run.smithi188.stdout: Birth: 2024-04-25 03:17:38.167337702 +0000 2024-04-25T03:22:06.504 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T03:22:06.606 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records in 2024-04-25T03:22:06.606 INFO:teuthology.orchestra.run.smithi188.stderr:1+0 records out 2024-04-25T03:22:06.606 INFO:teuthology.orchestra.run.smithi188.stderr:512 bytes copied, 0.000320514 s, 1.6 MB/s 2024-04-25T03:22:06.609 DEBUG:teuthology.orchestra.run.smithi188:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T03:22:06.654 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:22:06.654 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-25T03:22:06.655 INFO:tasks.ceph:Generating config... 2024-04-25T03:22:06.656 INFO:tasks.ceph:[client] rbd default map options = ms_mode=crc,rxbounce 2024-04-25T03:22:06.656 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-25T03:22:06.656 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-25T03:22:06.656 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T03:22:06.656 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T03:22:06.656 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T03:22:06.656 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T03:22:06.657 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T03:22:06.657 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T03:22:06.658 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T03:22:06.658 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-04-25T03:22:06.658 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-04-25T03:22:06.787 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/cluster2.keyring 2024-04-25T03:22:06.792 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-04-25T03:22:06.889 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-25T03:22:06.956 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.188')] 2024-04-25T03:22:06.956 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.188', 'mon warn on pool no app': False, 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default map options': 'ms_mode=crc,rxbounce'}, 'mon.a': {}} 2024-04-25T03:22:06.957 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:06.957 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T03:22:07.000 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.188 --print /home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:07.122 INFO:teuthology.orchestra.run.smithi188.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:monmaptool: generated fsid deb0a81e-61a8-4bcb-b28f-c9ae781a23c3 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:setting min_mon_release = quincy 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:epoch 0 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:fsid deb0a81e-61a8-4bcb-b28f-c9ae781a23c3 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:last_changed 2024-04-25T03:22:07.121532+0000 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:created 2024-04-25T03:22:07.121532+0000 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:min_mon_release 17 (quincy) 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:election_strategy: 1 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:0: [v2:172.21.15.188:3300/0,v1:172.21.15.188:6789/0] mon.a 2024-04-25T03:22:07.123 INFO:teuthology.orchestra.run.smithi188.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-04-25T03:22:07.126 DEBUG:teuthology.orchestra.run.smithi188:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T03:22:07.170 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID deb0a81e-61a8-4bcb-b28f-c9ae781a23c3... 2024-04-25T03:22:07.171 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-25T03:22:07.176 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-04-25T03:22:07.292 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2024-04-25T03:22:07.293 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2024-04-25T03:22:07.294 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2024-04-25T03:22:07.295 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = 172.21.15.188 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on pool no app = False 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on skipped message = False 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = deb0a81e-61a8-4bcb-b28f-c9ae781a23c3 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2024-04-25T03:22:07.296 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug trim objects = true 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.297 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: bdev async discard = True 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: bdev enable discard = True 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore allocator = bitmap 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 1/20 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 1/20 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 4/10 2024-04-25T03:22:07.298 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-04-25T03:22:07.299 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2024-04-25T03:22:07.300 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2024-04-25T03:22:07.301 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout: rbd default map options = "ms_mode=crc,rxbounce" 2024-04-25T03:22:07.302 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2024-04-25T03:22:07.321 INFO:teuthology.orchestra.run.smithi188.stdout:[global] 2024-04-25T03:22:07.321 INFO:teuthology.orchestra.run.smithi188.stdout: chdir = "" 2024-04-25T03:22:07.321 INFO:teuthology.orchestra.run.smithi188.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T03:22:07.321 INFO:teuthology.orchestra.run.smithi188.stdout: auth supported = cephx 2024-04-25T03:22:07.321 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: filestore xattr use omap = true 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: mon clock drift allowed = 1.000 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: osd crush chooseleaf type = 0 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: auth debug = true 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on old message = true 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on bug = true 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: mon pg warn max object skew = 0 2024-04-25T03:22:07.322 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default size = 2 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow primary affinity = true 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd allow pg remap = true 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on legacy crush tunables = false 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on crush straw calc version zero = false 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on no sortbitwise = false 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on osd down out interval zero = false 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on too few osds = false 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T03:22:07.323 INFO:teuthology.orchestra.run.smithi188.stdout: mon_allow_pool_size_one = true 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: osd default data pool replay window = 5 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: mon allow pool delete = true 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: mon cluster log file level = debug 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: debug asserts on shutdown = true 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: mon health detail to clog = false 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: mon host = 172.21.15.188 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: mon warn on pool no app = False 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: ms die on skipped message = False 2024-04-25T03:22:07.324 INFO:teuthology.orchestra.run.smithi188.stdout: fsid = deb0a81e-61a8-4bcb-b28f-c9ae781a23c3 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout:[osd] 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd journal size = 100 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub load threshold = 5.0 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd scrub max interval = 600 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd mclock profile = high_recovery_ops 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd recover clone overlap = true 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd recovery max chunk = 1048576 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug shutdown = true 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug op order = true 2024-04-25T03:22:07.325 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug verify stray on activate = true 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug trim objects = true 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: osd open classes on start = true 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug pg log writeout = true 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: osd deep scrub update digest min age = 30 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: osd map max advance = 10 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: journal zero on create = true 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: filestore ondisk finisher threads = 3 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: filestore apply finisher threads = 3 2024-04-25T03:22:07.326 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: bdev debug aio = true 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: osd debug misdirected ops = true 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: bdev async discard = True 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: bdev enable discard = True 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore allocator = bitmap 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore block size = 96636764160 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: bluestore fsck on mount = True 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluefs = 1/20 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: debug bluestore = 1/20 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: debug osd = 20 2024-04-25T03:22:07.327 INFO:teuthology.orchestra.run.smithi188.stdout: debug rocksdb = 4/10 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd full ratio = 0.9 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd nearfull ratio = 0.8 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: osd failsafe full ratio = 0.95 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: osd objectstore = bluestore 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr] 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: debug mgr = 20 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-04-25T03:22:07.328 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mgr/telemetry/nag = false 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout:[mon] 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: debug ms = 1 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: debug mon = 20 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: debug paxos = 20 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: debug auth = 20 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mon data avail warn = 5 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mon mgr mkfs grace = 240 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min pgs per osd = 4 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd reporter subtree level = osd 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mon osd prime pg temp = true 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: mon reweight min bytes per osd = 10 2024-04-25T03:22:07.329 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: # 1m isn't quite enough 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: mon_down_mkfs_grace = 2m 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.330 INFO:teuthology.orchestra.run.smithi188.stdout: mon_warn_on_filestore_osds = false 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout:[client] 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: rgw cache enabled = true 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable ops log = true 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: rgw enable usage log = true 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout: rbd default map options = "ms_mode=crc,rxbounce" 2024-04-25T03:22:07.331 INFO:teuthology.orchestra.run.smithi188.stdout:[mon.a] 2024-04-25T03:22:07.348 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-04-25T03:22:07.348 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-04-25T03:22:07.458 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T03:22:07.458 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:07.458 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-04-25T03:22:07.502 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:07.502 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-04-25T03:22:07.585 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T03:22:07.586 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:22:07.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-25T03:22:07.586 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-25T03:22:07.676 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T03:22:07.676 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:07.760 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T03:22:07.760 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:07.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/ceph/cluster2.keyring 2024-04-25T03:22:07.760 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-04-25T03:22:07.850 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:07.850 DEBUG:teuthology.orchestra.run.smithi188:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:07.936 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T03:22:07.936 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T03:22:07.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-25T03:22:08.093 INFO:teuthology.orchestra.run.smithi188.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-04-25T03:22:08.099 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T03:22:08.099 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T03:22:08.100 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-04-25T03:22:08.193 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-04-25T03:22:08.223 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-25T03:22:08.316 INFO:teuthology.orchestra.run.smithi188.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-04-25T03:22:08.345 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T03:22:08.345 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi188.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-25T03:22:08.345 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-04-25T03:22:08.412 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:22:08.412 INFO:tasks.ceph:role: cluster2.osd.0 2024-04-25T03:22:08.413 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T03:22:08.413 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T03:22:08.486 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T03:22:08.487 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T03:22:08.858 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-04-25T03:22:08.866 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-04-25T03:22:08.866 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-04-25T03:22:09.028 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-04-25T03:22:09.101 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-04-25T03:22:09.167 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:22:09.167 INFO:tasks.ceph:role: cluster2.osd.1 2024-04-25T03:22:09.167 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T03:22:09.167 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T03:22:09.241 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T03:22:09.241 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T03:22:09.241 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T03:22:09.241 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T03:22:09.241 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T03:22:09.241 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-04-25T03:22:09.242 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T03:22:09.242 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T03:22:09.242 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T03:22:09.242 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T03:22:09.594 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-04-25T03:22:09.597 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-04-25T03:22:09.598 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-04-25T03:22:09.707 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-04-25T03:22:09.779 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-04-25T03:22:09.845 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-04-25T03:22:09.846 INFO:tasks.ceph:role: cluster2.osd.2 2024-04-25T03:22:09.846 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T03:22:09.846 DEBUG:teuthology.orchestra.run.smithi188:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T03:22:09.919 INFO:teuthology.orchestra.run.smithi188.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T03:22:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T03:22:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T03:22:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T03:22:09.919 INFO:teuthology.orchestra.run.smithi188.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T03:22:09.919 INFO:teuthology.orchestra.run.smithi188.stdout: = sunit=0 swidth=0 blks 2024-04-25T03:22:09.919 INFO:teuthology.orchestra.run.smithi188.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T03:22:09.920 INFO:teuthology.orchestra.run.smithi188.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T03:22:09.920 INFO:teuthology.orchestra.run.smithi188.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T03:22:09.920 INFO:teuthology.orchestra.run.smithi188.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T03:22:10.293 INFO:teuthology.orchestra.run.smithi188.stdout:Discarding blocks...Done. 2024-04-25T03:22:10.297 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi188.front.sepia.ceph.com -o noatime 2024-04-25T03:22:10.297 DEBUG:teuthology.orchestra.run.smithi188:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-04-25T03:22:10.430 DEBUG:teuthology.orchestra.run.smithi188:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-04-25T03:22:10.504 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:10.623 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:10.620+0000 7f3ae461d7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-04-25T03:22:10.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:10.624+0000 7f3ae461d7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-04-25T03:22:10.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:10.624+0000 7f3ae461d7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-25T03:22:10.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:10.624+0000 7f3ae461d7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-25T03:22:10.627 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:10.624+0000 7f3ae461d7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-04-25T03:22:10.629 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:10.626+0000 7f3ae461d7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-04-25T03:22:13.562 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-04-25T03:22:13.640 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:13.743 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:13.740+0000 7f44b333e7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-04-25T03:22:13.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:13.741+0000 7f44b333e7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-04-25T03:22:13.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:13.741+0000 7f44b333e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-25T03:22:13.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:13.741+0000 7f44b333e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-25T03:22:13.745 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:13.741+0000 7f44b333e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-04-25T03:22:13.746 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:13.742+0000 7f44b333e7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-04-25T03:22:16.669 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-04-25T03:22:16.737 DEBUG:teuthology.orchestra.run.smithi188:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:16.842 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:16.840+0000 7fba5a1747c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-04-25T03:22:16.843 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:16.840+0000 7fba5a1747c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-04-25T03:22:16.843 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:16.840+0000 7fba5a1747c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-25T03:22:16.843 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:16.840+0000 7fba5a1747c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-25T03:22:16.843 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:16.840+0000 7fba5a1747c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-04-25T03:22:16.844 INFO:teuthology.orchestra.run.smithi188.stderr:2024-04-25T03:22:16.842+0000 7fba5a1747c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-04-25T03:22:19.755 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-04-25T03:22:19.821 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T03:22:19.822 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:19.822 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-04-25T03:22:19.887 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:19.887 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-04-25T03:22:19.993 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:19.993 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-04-25T03:22:20.099 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:20.099 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-04-25T03:22:20.205 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:20.205 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-04-25T03:22:20.289 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:20.289 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-04-25T03:22:20.372 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T03:22:20.372 DEBUG:teuthology.orchestra.run.smithi188:> sudo tee -a /etc/ceph/cluster2.keyring 2024-04-25T03:22:20.476 INFO:teuthology.orchestra.run.smithi188.stdout:[mgr.x] 2024-04-25T03:22:20.476 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBgzClmzkmABRAA+B0StJlkzmHX1/6Cz8D24Q== 2024-04-25T03:22:20.476 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.0] 2024-04-25T03:22:20.476 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBizClmnkcTJRAAwcP/JRSZhcf6LYEQfbyC7Q== 2024-04-25T03:22:20.476 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.1] 2024-04-25T03:22:20.476 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBlzClm5PM7LBAA3H5hex+9KJdvhuuPfUZ6qw== 2024-04-25T03:22:20.477 INFO:teuthology.orchestra.run.smithi188.stdout:[osd.2] 2024-04-25T03:22:20.477 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBozClmR3QgMhAAxTKiTmLW6KSnYqROVW2n0Q== 2024-04-25T03:22:20.477 INFO:teuthology.orchestra.run.smithi188.stdout:[client.mirror] 2024-04-25T03:22:20.477 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBgzClmxdB4CxAAWMOmCqkrWZQI+FQ2Uc/crA== 2024-04-25T03:22:20.477 INFO:teuthology.orchestra.run.smithi188.stdout:[client.mirror.0] 2024-04-25T03:22:20.477 INFO:teuthology.orchestra.run.smithi188.stdout: key = AQBgzClmJA7IEhAADxtbdpEqPK/xe9oSrPZXmg== 2024-04-25T03:22:20.479 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T03:22:20.626 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T03:22:20.732 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T03:22:20.839 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T03:22:20.946 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T03:22:21.056 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T03:22:21.165 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T03:22:21.165 DEBUG:teuthology.orchestra.run.smithi188:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-04-25T03:22:21.231 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-04-25T03:22:21.352 DEBUG:teuthology.orchestra.run.smithi188:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-04-25T03:22:21.418 DEBUG:teuthology.orchestra.run.smithi188:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-04-25T03:22:21.461 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-04-25T03:22:21.462 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-04-25T03:22:21.462 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-04-25T03:22:21.505 INFO:tasks.ceph.cluster2.mon.a:Started 2024-04-25T03:22:21.506 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-04-25T03:22:21.506 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-04-25T03:22:21.506 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-04-25T03:22:21.509 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-04-25T03:22:21.510 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T03:22:21.510 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster cluster2 osd crush tunables default 2024-04-25T03:22:21.793 INFO:teuthology.orchestra.run.smithi188.stderr:adjusted tunables profile to default 2024-04-25T03:22:21.809 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T03:22:21.809 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-04-25T03:22:21.809 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:21.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-04-25T03:22:21.877 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:21.878 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-04-25T03:22:21.984 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:22:21.985 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-04-25T03:22:22.012 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:22.010+0000 7f41984c5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T03:22:22.078 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster cluster2 osd new d6d5e21d-e5ed-4729-bed8-061304dc8eb8 0 2024-04-25T03:22:22.235 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:22.233+0000 7f41984c5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T03:22:22.475 INFO:teuthology.orchestra.run.smithi188.stdout:0 2024-04-25T03:22:22.493 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster cluster2 osd new ad829940-0ef2-4ba3-acf5-58a3f5149282 1 2024-04-25T03:22:22.691 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:22.689+0000 7f41984c5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T03:22:22.781 INFO:teuthology.orchestra.run.smithi188.stdout:1 2024-04-25T03:22:22.800 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster cluster2 osd new eb229790-f353-4d08-be57-b1bffedf8966 2 2024-04-25T03:22:23.086 INFO:teuthology.orchestra.run.smithi188.stdout:2 2024-04-25T03:22:23.104 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-04-25T03:22:23.105 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-04-25T03:22:23.108 INFO:tasks.ceph.cluster2.osd.0:Started 2024-04-25T03:22:23.108 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-04-25T03:22:23.109 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-04-25T03:22:23.112 INFO:tasks.ceph.cluster2.osd.1:Started 2024-04-25T03:22:23.112 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-04-25T03:22:23.112 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-04-25T03:22:23.117 INFO:tasks.ceph.cluster2.osd.2:Started 2024-04-25T03:22:23.117 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-25T03:22:23.391 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:23.389+0000 7f41984c5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T03:22:23.442 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:23.442 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":5,"fsid":"deb0a81e-61a8-4bcb-b28f-c9ae781a23c3","created":"2024-04-25T03:22:21.673300+0000","modified":"2024-04-25T03:22:23.083689+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T03:22:23.083689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6d5e21d-e5ed-4729-bed8-061304dc8eb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad829940-0ef2-4ba3-acf5-58a3f5149282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb229790-f353-4d08-be57-b1bffedf8966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:23.460 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-04-25T03:22:23.461 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T03:22:23.656 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:23.654+0000 7f41984c5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T03:22:23.763 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-25T03:22:23.796 INFO:tasks.ceph.cluster2.osd.2.smithi188.stderr:2024-04-25T03:22:23.794+0000 7fe3b6def7c0 -1 Falling back to public interface 2024-04-25T03:22:23.818 INFO:tasks.ceph.cluster2.osd.1.smithi188.stderr:2024-04-25T03:22:23.815+0000 7fdf58eea7c0 -1 Falling back to public interface 2024-04-25T03:22:23.827 INFO:tasks.ceph.cluster2.osd.0.smithi188.stderr:2024-04-25T03:22:23.825+0000 7f3e333687c0 -1 Falling back to public interface 2024-04-25T03:22:24.034 INFO:teuthology.misc.health.smithi188.stdout: 2024-04-25T03:22:24.034 INFO:teuthology.misc.health.smithi188.stdout:{"epoch":5,"fsid":"deb0a81e-61a8-4bcb-b28f-c9ae781a23c3","created":"2024-04-25T03:22:21.673300+0000","modified":"2024-04-25T03:22:23.083689+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T03:22:23.083689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6d5e21d-e5ed-4729-bed8-061304dc8eb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad829940-0ef2-4ba3-acf5-58a3f5149282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb229790-f353-4d08-be57-b1bffedf8966","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:24.051 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-25T03:22:24.147 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:24.145+0000 7f41984c5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T03:22:24.991 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:24.989+0000 7f41984c5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T03:22:25.086 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:25.084+0000 7f41984c5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T03:22:25.255 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:25.253+0000 7f41984c5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T03:22:25.471 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:25.469+0000 7f41984c5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T03:22:25.569 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:25.567+0000 7f41984c5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T03:22:25.669 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:25.667+0000 7f41984c5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T03:22:25.768 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:25.766+0000 7f41984c5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T03:22:25.883 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:25.881+0000 7f41984c5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T03:22:26.291 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:26.289+0000 7f41984c5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T03:22:26.407 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:26.405+0000 7f41984c5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T03:22:26.610 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:26.608+0000 7f41984c5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T03:22:26.719 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:26.717+0000 7f41984c5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T03:22:27.427 INFO:tasks.ceph.cluster2.osd.2.smithi188.stderr:2024-04-25T03:22:27.425+0000 7fe3b6def7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T03:22:27.483 INFO:tasks.ceph.cluster2.osd.1.smithi188.stderr:2024-04-25T03:22:27.481+0000 7fdf58eea7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T03:22:27.485 INFO:tasks.ceph.cluster2.osd.0.smithi188.stderr:2024-04-25T03:22:27.483+0000 7f3e333687c0 -1 osd.0 0 log_to_monitors true 2024-04-25T03:22:27.607 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:27.605+0000 7f41984c5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T03:22:27.816 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:27.814+0000 7f41984c5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T03:22:28.019 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:28.016+0000 7f41984c5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T03:22:28.525 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:28.521+0000 7f41984c5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T03:22:28.769 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:28.767+0000 7f41984c5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T03:22:29.303 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:29.301+0000 7f41984c5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T03:22:29.459 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:29.457+0000 7f41984c5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T03:22:29.606 INFO:tasks.ceph.cluster2.osd.2.smithi188.stderr:2024-04-25T03:22:29.604+0000 7fe3a4000700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T03:22:29.633 INFO:tasks.ceph.cluster2.osd.2.smithi188.stderr:2024-04-25T03:22:29.622+0000 7fe39a200700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T03:22:30.354 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-25T03:22:30.481 INFO:tasks.ceph.cluster2.mgr.x.smithi188.stderr:2024-04-25T03:22:30.479+0000 7f414e600700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-25T03:22:30.986 INFO:tasks.ceph.cluster2.osd.0.smithi188.stderr:2024-04-25T03:22:30.984+0000 7f3e20600700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T03:22:30.987 INFO:teuthology.misc.health.smithi188.stdout: 2024-04-25T03:22:30.987 INFO:teuthology.misc.health.smithi188.stdout:{"epoch":9,"fsid":"deb0a81e-61a8-4bcb-b28f-c9ae781a23c3","created":"2024-04-25T03:22:21.673300+0000","modified":"2024-04-25T03:22:30.606160+0000","last_up_change":"2024-04-25T03:22:30.606160+0000","last_in_change":"2024-04-25T03:22:23.083689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d6d5e21d-e5ed-4729-bed8-061304dc8eb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ad829940-0ef2-4ba3-acf5-58a3f5149282","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb229790-f353-4d08-be57-b1bffedf8966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6801","nonce":840963968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6803","nonce":840963968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6807","nonce":840963968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6805","nonce":840963968}]},"public_addr":"172.21.15.188:6801/840963968","cluster_addr":"172.21.15.188:6803/840963968","heartbeat_back_addr":"172.21.15.188:6807/840963968","heartbeat_front_addr":"172.21.15.188:6805/840963968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:30.992 INFO:tasks.ceph.cluster2.osd.0.smithi188.stderr:2024-04-25T03:22:30.990+0000 7f3e16800700 -1 osd.0 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T03:22:30.999 INFO:tasks.ceph.cluster2.osd.1.smithi188.stderr:2024-04-25T03:22:30.997+0000 7fdf48000700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T03:22:31.008 DEBUG:teuthology.misc:1 of 3 OSDs are up 2024-04-25T03:22:31.013 INFO:tasks.ceph.cluster2.osd.1.smithi188.stderr:2024-04-25T03:22:31.011+0000 7fdf3c400700 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T03:22:37.311 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-04-25T03:22:37.712 INFO:teuthology.misc.health.smithi188.stdout: 2024-04-25T03:22:37.712 INFO:teuthology.misc.health.smithi188.stdout:{"epoch":14,"fsid":"deb0a81e-61a8-4bcb-b28f-c9ae781a23c3","created":"2024-04-25T03:22:21.673300+0000","modified":"2024-04-25T03:22:36.008490+0000","last_up_change":"2024-04-25T03:22:31.986774+0000","last_in_change":"2024-04-25T03:22:23.083689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:22:33.530774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6d5e21d-e5ed-4729-bed8-061304dc8eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6817","nonce":3218143884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6819","nonce":3218143884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6823","nonce":3218143884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6821","nonce":3218143884}]},"public_addr":"172.21.15.188:6817/3218143884","cluster_addr":"172.21.15.188:6819/3218143884","heartbeat_back_addr":"172.21.15.188:6823/3218143884","heartbeat_front_addr":"172.21.15.188:6821/3218143884","state":["exists","up"]},{"osd":1,"uuid":"ad829940-0ef2-4ba3-acf5-58a3f5149282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6809","nonce":3431739252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6811","nonce":3431739252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6815","nonce":3431739252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6813","nonce":3431739252}]},"public_addr":"172.21.15.188:6809/3431739252","cluster_addr":"172.21.15.188:6811/3431739252","heartbeat_back_addr":"172.21.15.188:6815/3431739252","heartbeat_front_addr":"172.21.15.188:6813/3431739252","state":["exists","up"]},{"osd":2,"uuid":"eb229790-f353-4d08-be57-b1bffedf8966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6801","nonce":840963968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6803","nonce":840963968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6807","nonce":840963968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6805","nonce":840963968}]},"public_addr":"172.21.15.188:6801/840963968","cluster_addr":"172.21.15.188:6803/840963968","heartbeat_back_addr":"172.21.15.188:6807/840963968","heartbeat_front_addr":"172.21.15.188:6805/840963968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.458734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.452573+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.448465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:37.730 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-25T03:22:37.730 INFO:tasks.ceph:Creating RBD pool 2024-04-25T03:22:37.730 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-04-25T03:22:39.031 INFO:teuthology.orchestra.run.smithi188.stderr:pool 'rbd' created 2024-04-25T03:22:39.051 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-25T03:22:40.034 INFO:teuthology.orchestra.run.smithi188.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-25T03:22:40.056 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-04-25T03:22:40.056 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-04-25T03:22:40.056 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-04-25T03:22:40.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-04-25T03:22:40.057 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T03:22:40.529 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:40.548 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6824","nonce":1884378084},{"type":"v1","addr":"172.21.15.188:6825","nonce":1884378084}]},"active_addr":"172.21.15.188:6825/1884378084","active_change":"2024-04-25T03:22:29.459897+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":1223360810}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":1222363841}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":4202218771}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.188:0","nonce":2936556861}]}]} 2024-04-25T03:22:40.552 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-04-25T03:22:40.552 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-04-25T03:22:40.552 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-25T03:22:40.963 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:40.964 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":16,"fsid":"deb0a81e-61a8-4bcb-b28f-c9ae781a23c3","created":"2024-04-25T03:22:21.673300+0000","modified":"2024-04-25T03:22:40.022415+0000","last_up_change":"2024-04-25T03:22:31.986774+0000","last_in_change":"2024-04-25T03:22:23.083689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:22:33.530774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T03:22:38.170609+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6d5e21d-e5ed-4729-bed8-061304dc8eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6817","nonce":3218143884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6819","nonce":3218143884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6823","nonce":3218143884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6821","nonce":3218143884}]},"public_addr":"172.21.15.188:6817/3218143884","cluster_addr":"172.21.15.188:6819/3218143884","heartbeat_back_addr":"172.21.15.188:6823/3218143884","heartbeat_front_addr":"172.21.15.188:6821/3218143884","state":["exists","up"]},{"osd":1,"uuid":"ad829940-0ef2-4ba3-acf5-58a3f5149282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6809","nonce":3431739252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6811","nonce":3431739252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6815","nonce":3431739252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6813","nonce":3431739252}]},"public_addr":"172.21.15.188:6809/3431739252","cluster_addr":"172.21.15.188:6811/3431739252","heartbeat_back_addr":"172.21.15.188:6815/3431739252","heartbeat_front_addr":"172.21.15.188:6813/3431739252","state":["exists","up"]},{"osd":2,"uuid":"eb229790-f353-4d08-be57-b1bffedf8966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6801","nonce":840963968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6803","nonce":840963968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6807","nonce":840963968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6805","nonce":840963968}]},"public_addr":"172.21.15.188:6801/840963968","cluster_addr":"172.21.15.188:6803/840963968","heartbeat_back_addr":"172.21.15.188:6807/840963968","heartbeat_front_addr":"172.21.15.188:6805/840963968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.458734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.452573+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.448465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:40.984 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-04-25T03:22:40.984 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-25T03:22:41.394 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:41.395 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":17,"fsid":"deb0a81e-61a8-4bcb-b28f-c9ae781a23c3","created":"2024-04-25T03:22:21.673300+0000","modified":"2024-04-25T03:22:41.028135+0000","last_up_change":"2024-04-25T03:22:31.986774+0000","last_in_change":"2024-04-25T03:22:23.083689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:22:33.530774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T03:22:38.170609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6d5e21d-e5ed-4729-bed8-061304dc8eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6817","nonce":3218143884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6819","nonce":3218143884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6823","nonce":3218143884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6821","nonce":3218143884}]},"public_addr":"172.21.15.188:6817/3218143884","cluster_addr":"172.21.15.188:6819/3218143884","heartbeat_back_addr":"172.21.15.188:6823/3218143884","heartbeat_front_addr":"172.21.15.188:6821/3218143884","state":["exists","up"]},{"osd":1,"uuid":"ad829940-0ef2-4ba3-acf5-58a3f5149282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6809","nonce":3431739252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6811","nonce":3431739252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6815","nonce":3431739252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6813","nonce":3431739252}]},"public_addr":"172.21.15.188:6809/3431739252","cluster_addr":"172.21.15.188:6811/3431739252","heartbeat_back_addr":"172.21.15.188:6815/3431739252","heartbeat_front_addr":"172.21.15.188:6813/3431739252","state":["exists","up"]},{"osd":2,"uuid":"eb229790-f353-4d08-be57-b1bffedf8966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6801","nonce":840963968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6803","nonce":840963968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6807","nonce":840963968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6805","nonce":840963968}]},"public_addr":"172.21.15.188:6801/840963968","cluster_addr":"172.21.15.188:6803/840963968","heartbeat_back_addr":"172.21.15.188:6807/840963968","heartbeat_front_addr":"172.21.15.188:6805/840963968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.458734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.452573+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.448465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T03:22:41.416 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-04-25T03:22:41.416 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-04-25T03:22:41.416 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-04-25T03:22:41.670 INFO:teuthology.orchestra.run.smithi188.stdout:38654705668 2024-04-25T03:22:41.670 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-25T03:22:41.688 INFO:teuthology.orchestra.run.smithi188.stdout:42949672964 2024-04-25T03:22:41.688 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-25T03:22:41.692 INFO:teuthology.orchestra.run.smithi188.stdout:42949672964 2024-04-25T03:22:41.693 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-25T03:22:42.112 INFO:teuthology.orchestra.run.smithi188.stdout:38654705667 2024-04-25T03:22:42.133 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705667 for osd.2 2024-04-25T03:22:42.168 INFO:teuthology.orchestra.run.smithi188.stdout:42949672963 2024-04-25T03:22:42.173 INFO:teuthology.orchestra.run.smithi188.stdout:42949672963 2024-04-25T03:22:42.191 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672964 got 42949672963 for osd.0 2024-04-25T03:22:42.195 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672964 got 42949672963 for osd.1 2024-04-25T03:22:43.134 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-25T03:22:43.192 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-25T03:22:43.195 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-25T03:22:43.584 INFO:teuthology.orchestra.run.smithi188.stdout:38654705667 2024-04-25T03:22:43.607 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705667 for osd.2 2024-04-25T03:22:43.635 INFO:teuthology.orchestra.run.smithi188.stdout:42949672963 2024-04-25T03:22:43.645 INFO:teuthology.orchestra.run.smithi188.stdout:42949672963 2024-04-25T03:22:43.656 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672964 got 42949672963 for osd.0 2024-04-25T03:22:43.666 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672964 got 42949672963 for osd.1 2024-04-25T03:22:44.607 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-04-25T03:22:44.657 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-04-25T03:22:44.666 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-04-25T03:22:45.057 INFO:teuthology.orchestra.run.smithi188.stdout:38654705668 2024-04-25T03:22:45.080 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705668 got 38654705668 for osd.2 2024-04-25T03:22:45.080 DEBUG:teuthology.parallel:result is None 2024-04-25T03:22:45.101 INFO:teuthology.orchestra.run.smithi188.stdout:42949672965 2024-04-25T03:22:45.123 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672964 got 42949672965 for osd.0 2024-04-25T03:22:45.123 DEBUG:teuthology.parallel:result is None 2024-04-25T03:22:45.140 INFO:teuthology.orchestra.run.smithi188.stdout:42949672965 2024-04-25T03:22:45.160 INFO:tasks.ceph.ceph_manager.cluster2:need seq 42949672964 got 42949672965 for osd.1 2024-04-25T03:22:45.161 DEBUG:teuthology.parallel:result is None 2024-04-25T03:22:45.161 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-25T03:22:45.161 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-25T03:22:45.610 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:45.613 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-04-25T03:22:45.630 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-25T03:22:45.473567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82656,"kb_used_data":1584,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032864,"statfs":{"total":289910292480,"available":289825652736,"internally_reserved":0,"allocated":1622016,"data_stored":1300675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.450822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388919+0000","last_change":"2024-04-25T03:22:40.043916+0000","last_active":"2024-04-25T03:22:41.388919+0000","last_peered":"2024-04-25T03:22:41.388919+0000","last_clean":"2024-04-25T03:22:41.388919+0000","last_became_active":"2024-04-25T03:22:40.043563+0000","last_became_peered":"2024-04-25T03:22:40.043563+0000","last_unstale":"2024-04-25T03:22:41.388919+0000","last_undegraded":"2024-04-25T03:22:41.388919+0000","last_fullsized":"2024-04-25T03:22:41.388919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:20.327339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388926+0000","last_change":"2024-04-25T03:22:40.040386+0000","last_active":"2024-04-25T03:22:41.388926+0000","last_peered":"2024-04-25T03:22:41.388926+0000","last_clean":"2024-04-25T03:22:41.388926+0000","last_became_active":"2024-04-25T03:22:40.040101+0000","last_became_peered":"2024-04-25T03:22:40.040101+0000","last_unstale":"2024-04-25T03:22:41.388926+0000","last_undegraded":"2024-04-25T03:22:41.388926+0000","last_fullsized":"2024-04-25T03:22:41.388926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:26.847462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388946+0000","last_change":"2024-04-25T03:22:40.044306+0000","last_active":"2024-04-25T03:22:41.388946+0000","last_peered":"2024-04-25T03:22:41.388946+0000","last_clean":"2024-04-25T03:22:41.388946+0000","last_became_active":"2024-04-25T03:22:40.044067+0000","last_became_peered":"2024-04-25T03:22:40.044067+0000","last_unstale":"2024-04-25T03:22:41.388946+0000","last_undegraded":"2024-04-25T03:22:41.388946+0000","last_fullsized":"2024-04-25T03:22:41.388946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:15.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-25T03:22:40.042074+0000","last_change":"2024-04-25T03:22:40.041323+0000","last_active":"2024-04-25T03:22:40.042074+0000","last_peered":"2024-04-25T03:22:40.042074+0000","last_clean":"2024-04-25T03:22:40.042074+0000","last_became_active":"2024-04-25T03:22:40.040953+0000","last_became_peered":"2024-04-25T03:22:40.040953+0000","last_unstale":"2024-04-25T03:22:40.042074+0000","last_undegraded":"2024-04-25T03:22:40.042074+0000","last_fullsized":"2024-04-25T03:22:40.042074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:08.753900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":145,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388723+0000","last_change":"2024-04-25T03:22:35.092603+0000","last_active":"2024-04-25T03:22:41.388723+0000","last_peered":"2024-04-25T03:22:41.388723+0000","last_clean":"2024-04-25T03:22:41.388723+0000","last_became_active":"2024-04-25T03:22:35.092211+0000","last_became_peered":"2024-04-25T03:22:35.092211+0000","last_unstale":"2024-04-25T03:22:41.388723+0000","last_undegraded":"2024-04-25T03:22:41.388723+0000","last_fullsized":"2024-04-25T03:22:41.388723+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_clean_scrub_stamp":"2024-04-25T03:22:34.001265+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:44.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388868+0000","last_change":"2024-04-25T03:22:40.038756+0000","last_active":"2024-04-25T03:22:41.388868+0000","last_peered":"2024-04-25T03:22:41.388868+0000","last_clean":"2024-04-25T03:22:41.388868+0000","last_became_active":"2024-04-25T03:22:40.038422+0000","last_became_peered":"2024-04-25T03:22:40.038422+0000","last_unstale":"2024-04-25T03:22:41.388868+0000","last_undegraded":"2024-04-25T03:22:41.388868+0000","last_fullsized":"2024-04-25T03:22:41.388868+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:30.846205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-25T03:22:40.043391+0000","last_change":"2024-04-25T03:22:40.041941+0000","last_active":"2024-04-25T03:22:40.043391+0000","last_peered":"2024-04-25T03:22:40.043391+0000","last_clean":"2024-04-25T03:22:40.043391+0000","last_became_active":"2024-04-25T03:22:40.041506+0000","last_became_peered":"2024-04-25T03:22:40.041506+0000","last_unstale":"2024-04-25T03:22:40.043391+0000","last_undegraded":"2024-04-25T03:22:40.043391+0000","last_fullsized":"2024-04-25T03:22:40.043391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:21.938707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.033121+0000","last_change":"2024-04-25T03:22:40.041527+0000","last_active":"2024-04-25T03:22:41.033121+0000","last_peered":"2024-04-25T03:22:41.033121+0000","last_clean":"2024-04-25T03:22:41.033121+0000","last_became_active":"2024-04-25T03:22:40.041252+0000","last_became_peered":"2024-04-25T03:22:40.041252+0000","last_unstale":"2024-04-25T03:22:41.033121+0000","last_undegraded":"2024-04-25T03:22:41.033121+0000","last_fullsized":"2024-04-25T03:22:41.033121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.379837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388935+0000","last_change":"2024-04-25T03:22:40.043790+0000","last_active":"2024-04-25T03:22:41.388935+0000","last_peered":"2024-04-25T03:22:41.388935+0000","last_clean":"2024-04-25T03:22:41.388935+0000","last_became_active":"2024-04-25T03:22:40.043485+0000","last_became_peered":"2024-04-25T03:22:40.043485+0000","last_unstale":"2024-04-25T03:22:41.388935+0000","last_undegraded":"2024-04-25T03:22:41.388935+0000","last_fullsized":"2024-04-25T03:22:41.388935+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:16.009245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":741376,"data_stored":631507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":37661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":741376,"data_stored":631507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T03:22:45.630 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-25T03:22:46.036 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:46.039 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-04-25T03:22:46.056 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-25T03:22:45.473567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82656,"kb_used_data":1584,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032864,"statfs":{"total":289910292480,"available":289825652736,"internally_reserved":0,"allocated":1622016,"data_stored":1300675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.450822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388919+0000","last_change":"2024-04-25T03:22:40.043916+0000","last_active":"2024-04-25T03:22:41.388919+0000","last_peered":"2024-04-25T03:22:41.388919+0000","last_clean":"2024-04-25T03:22:41.388919+0000","last_became_active":"2024-04-25T03:22:40.043563+0000","last_became_peered":"2024-04-25T03:22:40.043563+0000","last_unstale":"2024-04-25T03:22:41.388919+0000","last_undegraded":"2024-04-25T03:22:41.388919+0000","last_fullsized":"2024-04-25T03:22:41.388919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:20.327339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388926+0000","last_change":"2024-04-25T03:22:40.040386+0000","last_active":"2024-04-25T03:22:41.388926+0000","last_peered":"2024-04-25T03:22:41.388926+0000","last_clean":"2024-04-25T03:22:41.388926+0000","last_became_active":"2024-04-25T03:22:40.040101+0000","last_became_peered":"2024-04-25T03:22:40.040101+0000","last_unstale":"2024-04-25T03:22:41.388926+0000","last_undegraded":"2024-04-25T03:22:41.388926+0000","last_fullsized":"2024-04-25T03:22:41.388926+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:26.847462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388946+0000","last_change":"2024-04-25T03:22:40.044306+0000","last_active":"2024-04-25T03:22:41.388946+0000","last_peered":"2024-04-25T03:22:41.388946+0000","last_clean":"2024-04-25T03:22:41.388946+0000","last_became_active":"2024-04-25T03:22:40.044067+0000","last_became_peered":"2024-04-25T03:22:40.044067+0000","last_unstale":"2024-04-25T03:22:41.388946+0000","last_undegraded":"2024-04-25T03:22:41.388946+0000","last_fullsized":"2024-04-25T03:22:41.388946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:15.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-25T03:22:40.042074+0000","last_change":"2024-04-25T03:22:40.041323+0000","last_active":"2024-04-25T03:22:40.042074+0000","last_peered":"2024-04-25T03:22:40.042074+0000","last_clean":"2024-04-25T03:22:40.042074+0000","last_became_active":"2024-04-25T03:22:40.040953+0000","last_became_peered":"2024-04-25T03:22:40.040953+0000","last_unstale":"2024-04-25T03:22:40.042074+0000","last_undegraded":"2024-04-25T03:22:40.042074+0000","last_fullsized":"2024-04-25T03:22:40.042074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:57:08.753900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":145,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388723+0000","last_change":"2024-04-25T03:22:35.092603+0000","last_active":"2024-04-25T03:22:41.388723+0000","last_peered":"2024-04-25T03:22:41.388723+0000","last_clean":"2024-04-25T03:22:41.388723+0000","last_became_active":"2024-04-25T03:22:35.092211+0000","last_became_peered":"2024-04-25T03:22:35.092211+0000","last_unstale":"2024-04-25T03:22:41.388723+0000","last_undegraded":"2024-04-25T03:22:41.388723+0000","last_fullsized":"2024-04-25T03:22:41.388723+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_clean_scrub_stamp":"2024-04-25T03:22:34.001265+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:44.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388868+0000","last_change":"2024-04-25T03:22:40.038756+0000","last_active":"2024-04-25T03:22:41.388868+0000","last_peered":"2024-04-25T03:22:41.388868+0000","last_clean":"2024-04-25T03:22:41.388868+0000","last_became_active":"2024-04-25T03:22:40.038422+0000","last_became_peered":"2024-04-25T03:22:40.038422+0000","last_unstale":"2024-04-25T03:22:41.388868+0000","last_undegraded":"2024-04-25T03:22:41.388868+0000","last_fullsized":"2024-04-25T03:22:41.388868+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:30.846205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-04-25T03:22:40.043391+0000","last_change":"2024-04-25T03:22:40.041941+0000","last_active":"2024-04-25T03:22:40.043391+0000","last_peered":"2024-04-25T03:22:40.043391+0000","last_clean":"2024-04-25T03:22:40.043391+0000","last_became_active":"2024-04-25T03:22:40.041506+0000","last_became_peered":"2024-04-25T03:22:40.041506+0000","last_unstale":"2024-04-25T03:22:40.043391+0000","last_undegraded":"2024-04-25T03:22:40.043391+0000","last_fullsized":"2024-04-25T03:22:40.043391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:21.938707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.033121+0000","last_change":"2024-04-25T03:22:40.041527+0000","last_active":"2024-04-25T03:22:41.033121+0000","last_peered":"2024-04-25T03:22:41.033121+0000","last_clean":"2024-04-25T03:22:41.033121+0000","last_became_active":"2024-04-25T03:22:40.041252+0000","last_became_peered":"2024-04-25T03:22:40.041252+0000","last_unstale":"2024-04-25T03:22:41.033121+0000","last_undegraded":"2024-04-25T03:22:41.033121+0000","last_fullsized":"2024-04-25T03:22:41.033121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.379837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-25T03:22:41.388935+0000","last_change":"2024-04-25T03:22:40.043790+0000","last_active":"2024-04-25T03:22:41.388935+0000","last_peered":"2024-04-25T03:22:41.388935+0000","last_clean":"2024-04-25T03:22:41.388935+0000","last_became_active":"2024-04-25T03:22:40.043485+0000","last_became_peered":"2024-04-25T03:22:40.043485+0000","last_unstale":"2024-04-25T03:22:41.388935+0000","last_undegraded":"2024-04-25T03:22:41.388935+0000","last_fullsized":"2024-04-25T03:22:41.388935+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:16.009245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":741376,"data_stored":631507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":139264,"data_stored":37661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27748,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344092,"statfs":{"total":96636764160,"available":96608350208,"internally_reserved":0,"allocated":741376,"data_stored":631507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T03:22:46.056 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-25T03:22:46.057 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-04-25T03:22:46.057 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-04-25T03:22:46.057 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-04-25T03:22:46.509 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T03:22:46.510 INFO:teuthology.orchestra.run.smithi188.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T03:22:46.530 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-04-25T03:22:46.530 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-25T03:22:46.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-04-25T03:22:46.563 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-04-25T03:22:46.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-04-25T03:22:46.567 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-04-25T03:22:46.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-04-25T03:22:46.587 DEBUG:teuthology.orchestra.run.smithi188:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-04-25T03:22:46.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-04-25T03:22:46.591 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T03:22:46.624 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T03:22:46.624 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T03:22:46.624 DEBUG:teuthology.orchestra.run.smithi188:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-25T03:22:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T03:22:46.669 INFO:teuthology.orchestra.run.smithi188.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-04-25T03:22:46.669 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-25T03:22:46.754 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-25T03:22:46.755 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-04-25T03:22:46.839 INFO:tasks.workunit:timeout=3h 2024-04-25T03:22:46.839 INFO:tasks.workunit:cleanup=True 2024-04-25T03:22:46.839 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T03:22:46.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-04-25T03:23:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Updating files: 66% (8361/12492) Updating files: 67% (8370/12492) Updating files: 68% (8495/12492) Updating files: 69% (8620/12492) Updating files: 70% (8745/12492) Updating files: 71% (8870/12492) Updating files: 72% (8995/12492) Updating files: 73% (9120/12492) Updating files: 74% (9245/12492) Updating files: 75% (9369/12492) Updating files: 76% (9494/12492) Updating files: 77% (9619/12492) Updating files: 78% (9744/12492) Updating files: 79% (9869/12492) Updating files: 80% (9994/12492) Updating files: 81% (10119/12492) Updating files: 82% (10244/12492) Updating files: 83% (10369/12492) Updating files: 84% (10494/12492) Updating files: 85% (10619/12492) Updating files: 86% (10744/12492) Updating files: 87% (10869/12492) Updating files: 88% (10993/12492) Updating files: 89% (11118/12492) Updating files: 90% (11243/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T03:23:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T03:23:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:23:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T03:23:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: git switch -c 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Or undo this operation with: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: git switch - 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:23:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T03:23:58.157 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-04-25T03:23:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T03:23:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-25T03:23:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T03:23:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T03:23:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T03:23:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-04-25T03:23:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-25T03:23:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T03:23:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-04-25T03:23:59.267 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-04-25T03:23:59.267 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-04-25T03:23:59.312 INFO:tasks.workunit:Running workunits matching rbd/compare_mirror_images.sh on cluster1.client.mirror... 2024-04-25T03:23:59.313 INFO:tasks.workunit:Running workunit rbd/compare_mirror_images.sh... 2024-04-25T03:23:59.313 DEBUG:teuthology.orchestra.run.smithi188:workunit test rbd/compare_mirror_images.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=krbd RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG_PREFIX=image-primary 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MIRROR_IMAGE_MODE=snapshot 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MIRROR_POOL_MODE=image 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT_PREFIX=test-primary 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RBD_MIRROR_INSTANCES=1 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RBD_MIRROR_MODE=snapshot 2024-04-25T03:23:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RBD_MIRROR_USE_EXISTING_CLUSTER=1 2024-04-25T03:23:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-04-25T03:23:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-04-25T03:23:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ type xmlstarlet 2024-04-25T03:23:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ XMLSTARLET=xmlstarlet 2024-04-25T03:23:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ RBD_MIRROR_INSTANCES=1 2024-04-25T03:23:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CLUSTER1=cluster1 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CLUSTER2=cluster2 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ PEER_CLUSTER_SUFFIX= 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ POOL=mirror 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ PARENT_POOL=mirror_parent 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ NS1=ns1 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ NS2=ns2 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ TEMPDIR= 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ID=mirror 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ RBD_MIRROR_MODE=snapshot 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ MIRROR_POOL_MODE=image 2024-04-25T03:23:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ '[' snapshot = snapshot ']' 2024-04-25T03:23:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ MIRROR_POOL_MODE=image 2024-04-25T03:23:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-04-25T03:23:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS='--id mirror' 2024-04-25T03:23:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ LAST_MIRROR_INSTANCE=0 2024-04-25T03:23:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-04-25T03:23:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-04-25T03:23:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-04-25T03:23:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_BIN=. 2024-04-25T03:23:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_SRC=. 2024-04-25T03:23:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ '[' -e CMakeCache.txt ']' 2024-04-25T03:23:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ '[' 0 -gt 0 ']' 2024-04-25T03:23:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ setup 2024-04-25T03:23:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local c 2024-04-25T03:23:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-04-25T03:23:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ setup_tempdir 2024-04-25T03:23:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:23:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ mktemp -d 2024-04-25T03:23:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ TEMPDIR=/tmp/tmp.gg15hFy7Os 2024-04-25T03:23:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -z 1 ']' 2024-04-25T03:23:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ setup_pools cluster1 cluster2 2024-04-25T03:23:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:23:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local remote_cluster=cluster2 2024-04-25T03:23:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mon_map_file 2024-04-25T03:23:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mon_addr 2024-04-25T03:23:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local admin_key_file 2024-04-25T03:23:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local uuid 2024-04-25T03:23:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:23:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-04-25T03:24:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror' already exists 2024-04-25T03:24:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-04-25T03:24:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror_parent' already exists 2024-04-25T03:24:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 pool init mirror 2024-04-25T03:24:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-04-25T03:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-04-25T03:24:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-04-25T03:24:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-04-25T03:24:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-04-25T03:24:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-04-25T03:24:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-04-25T03:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -z ']' 2024-04-25T03:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -z ']' 2024-04-25T03:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_add cluster1 mirror cluster2 2024-04-25T03:24:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local client_cluster=cluster2 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local remote_cluster=cluster2 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local uuid_var_name 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local error_code 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local peer_uuid 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 16 32 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set +e 2024-04-25T03:24:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_uuid=2430a790-08d7-45d5-83bb-6bb4efb2a2ae 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ error_code=0 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set -e 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 -eq 17 ']' 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test 0 -eq 0 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror_parent 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local client_cluster=cluster2 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local remote_cluster=cluster2 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local uuid_var_name 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local error_code 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local peer_uuid 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 16 32 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set +e 2024-04-25T03:24:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-04-25T03:24:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_uuid=82a0a23b-bc9c-4606-9fa5-21a861a21388 2024-04-25T03:24:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ error_code=0 2024-04-25T03:24:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set -e 2024-04-25T03:24:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 -eq 17 ']' 2024-04-25T03:24:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test 0 -eq 0 2024-04-25T03:24:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ setup_pools cluster2 cluster1 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local remote_cluster=cluster1 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mon_map_file 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mon_addr 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local admin_key_file 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local uuid 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-04-25T03:24:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror' already exists 2024-04-25T03:24:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-04-25T03:24:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror_parent' already exists 2024-04-25T03:24:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 pool init mirror 2024-04-25T03:24:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-04-25T03:24:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T03:24:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-04-25T03:24:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-04-25T03:24:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-04-25T03:24:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-04-25T03:24:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-04-25T03:24:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-04-25T03:24:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -z ']' 2024-04-25T03:24:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -z ']' 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_add cluster2 mirror cluster1 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local client_cluster=cluster1 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local remote_cluster=cluster1 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local uuid_var_name 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local error_code 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local peer_uuid 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 16 32 2024-04-25T03:24:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set +e 2024-04-25T03:24:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_uuid=64523b28-2a05-4e45-b6f3-ff92a209d930 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ error_code=0 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set -e 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 -eq 17 ']' 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test 0 -eq 0 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:24:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror_parent 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local client_cluster=cluster1 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local remote_cluster=cluster1 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local uuid_var_name 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local error_code 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local peer_uuid 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 16 32 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set +e 2024-04-25T03:24:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-04-25T03:24:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ peer_uuid=edf8df69-64ef-4054-808a-b096917415a6 2024-04-25T03:24:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ error_code=0 2024-04-25T03:24:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set -e 2024-04-25T03:24:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 -eq 17 ']' 2024-04-25T03:24:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test 0 -eq 0 2024-04-25T03:24:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:24:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ start_mirrors cluster1 2024-04-25T03:24:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ seq 0 0 2024-04-25T03:24:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ start_mirror cluster1:0 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1:0 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local instance 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local val=cluster1:0 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster_var_name=cluster 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local instance_var_name=instance 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cluster=cluster1 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ instance=0 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 = cluster1:0 ']' 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ eval cluster=cluster1 2024-04-25T03:24:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ cluster=cluster1 2024-04-25T03:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ eval instance=0 2024-04-25T03:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ instance=0 2024-04-25T03:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test -n 1 2024-04-25T03:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 2024-04-25T03:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ start_mirrors cluster2 2024-04-25T03:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:24:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ seq 0 0 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ start_mirror cluster2:0 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2:0 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local instance 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local val=cluster2:0 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster_var_name=cluster 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local instance_var_name=instance 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cluster=cluster2 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ instance=0 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 = cluster2:0 ']' 2024-04-25T03:24:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ eval cluster=cluster2 2024-04-25T03:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ cluster=cluster2 2024-04-25T03:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ eval instance=0 2024-04-25T03:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ instance=0 2024-04-25T03:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test -n 1 2024-04-25T03:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 2024-04-25T03:24:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wget https://download.ceph.com/qa/linux-5.4.tar.gz 2024-04-25T03:24:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:--2024-04-25 03:24:16-- https://download.ceph.com/qa/linux-5.4.tar.gz 2024-04-25T03:24:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-25T03:24:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:443... connected. 2024-04-25T03:24:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-25T03:24:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Length: 172616875 (165M) [application/octet-stream] 2024-04-25T03:24:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:Saving to: ‘linux-5.4.tar.gz’ 2024-04-25T03:24:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:24:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 0K .......... .......... .......... .......... .......... 0% 2.34M 70s 2024-04-25T03:24:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50K .......... .......... .......... .......... .......... 0% 2.38M 70s 2024-04-25T03:24:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 100K .......... .......... .......... .......... .......... 0% 189M 47s 2024-04-25T03:24:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 150K .......... .......... .......... .......... .......... 0% 2.40M 52s 2024-04-25T03:24:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 200K .......... .......... .......... .......... .......... 0% 164M 42s 2024-04-25T03:24:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 250K .......... .......... .......... .......... .......... 0% 179M 35s 2024-04-25T03:24:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 300K .......... .......... .......... .......... .......... 0% 188M 30s 2024-04-25T03:24:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 350K .......... .......... .......... .......... .......... 0% 2.44M 35s 2024-04-25T03:24:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 400K .......... .......... .......... .......... .......... 0% 184M 31s 2024-04-25T03:24:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 450K .......... .......... .......... .......... .......... 0% 158M 28s 2024-04-25T03:24:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 500K .......... .......... .......... .......... .......... 0% 178M 26s 2024-04-25T03:24:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 550K .......... .......... .......... .......... .......... 0% 180M 24s 2024-04-25T03:24:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 600K .......... .......... .......... .......... .......... 0% 2.54M 27s 2024-04-25T03:24:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 650K .......... .......... .......... .......... .......... 0% 130M 25s 2024-04-25T03:24:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 700K .......... .......... .......... .......... .......... 0% 190M 23s 2024-04-25T03:24:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 750K .......... .......... .......... .......... .......... 0% 151M 22s 2024-04-25T03:24:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 800K .......... .......... .......... .......... .......... 0% 174M 21s 2024-04-25T03:24:16.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 850K .......... .......... .......... .......... .......... 0% 180M 19s 2024-04-25T03:24:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 900K .......... .......... .......... .......... .......... 0% 2.62M 22s 2024-04-25T03:24:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 950K .......... .......... .......... .......... .......... 0% 91.4M 21s 2024-04-25T03:24:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1000K .......... .......... .......... .......... .......... 0% 146M 20s 2024-04-25T03:24:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1050K .......... .......... .......... .......... .......... 0% 180M 19s 2024-04-25T03:24:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1100K .......... .......... .......... .......... .......... 0% 182M 18s 2024-04-25T03:24:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1150K .......... .......... .......... .......... .......... 0% 171M 17s 2024-04-25T03:24:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1200K .......... .......... .......... .......... .......... 0% 184M 17s 2024-04-25T03:24:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1250K .......... .......... .......... .......... .......... 0% 2.54M 19s 2024-04-25T03:24:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1300K .......... .......... .......... .......... .......... 0% 135M 18s 2024-04-25T03:24:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1350K .......... .......... .......... .......... .......... 0% 145M 17s 2024-04-25T03:24:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1400K .......... .......... .......... .......... .......... 0% 175M 17s 2024-04-25T03:24:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1450K .......... .......... .......... .......... .......... 0% 189M 16s 2024-04-25T03:24:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1500K .......... .......... .......... .......... .......... 0% 194M 16s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1550K .......... .......... .......... .......... .......... 0% 2.67M 17s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1600K .......... .......... .......... .......... .......... 0% 117M 17s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1650K .......... .......... .......... .......... .......... 1% 122M 16s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1700K .......... .......... .......... .......... .......... 1% 148M 16s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1750K .......... .......... .......... .......... .......... 1% 149M 15s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1800K .......... .......... .......... .......... .......... 1% 185M 15s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1850K .......... .......... .......... .......... .......... 1% 160M 15s 2024-04-25T03:24:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1900K .......... .......... .......... .......... .......... 1% 194M 14s 2024-04-25T03:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 1950K .......... .......... .......... .......... .......... 1% 2.57M 15s 2024-04-25T03:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2000K .......... .......... .......... .......... .......... 1% 176M 15s 2024-04-25T03:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2050K .......... .......... .......... .......... .......... 1% 186M 15s 2024-04-25T03:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2100K .......... .......... .......... .......... .......... 1% 159M 14s 2024-04-25T03:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2150K .......... .......... .......... .......... .......... 1% 160M 14s 2024-04-25T03:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2200K .......... .......... .......... .......... .......... 1% 190M 14s 2024-04-25T03:24:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2250K .......... .......... .......... .......... .......... 1% 154M 14s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2300K .......... .......... .......... .......... .......... 1% 2.65M 15s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2350K .......... .......... .......... .......... .......... 1% 89.4M 14s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2400K .......... .......... .......... .......... .......... 1% 170M 14s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2450K .......... .......... .......... .......... .......... 1% 187M 14s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2500K .......... .......... .......... .......... .......... 1% 191M 13s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2550K .......... .......... .......... .......... .......... 1% 161M 13s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2600K .......... .......... .......... .......... .......... 1% 165M 13s 2024-04-25T03:24:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2650K .......... .......... .......... .......... .......... 1% 91.7M 13s 2024-04-25T03:24:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2700K .......... .......... .......... .......... .......... 1% 2.64M 14s 2024-04-25T03:24:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2750K .......... .......... .......... .......... .......... 1% 77.8M 13s 2024-04-25T03:24:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2800K .......... .......... .......... .......... .......... 1% 163M 13s 2024-04-25T03:24:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2850K .......... .......... .......... .......... .......... 1% 134M 13s 2024-04-25T03:24:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2900K .......... .......... .......... .......... .......... 1% 133M 13s 2024-04-25T03:24:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2950K .......... .......... .......... .......... .......... 1% 126M 13s 2024-04-25T03:24:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3000K .......... .......... .......... .......... .......... 1% 160M 12s 2024-04-25T03:24:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3050K .......... .......... .......... .......... .......... 1% 117M 12s 2024-04-25T03:24:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3100K .......... .......... .......... .......... .......... 1% 2.80M 13s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3150K .......... .......... .......... .......... .......... 1% 68.1M 13s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3200K .......... .......... .......... .......... .......... 1% 143M 13s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3250K .......... .......... .......... .......... .......... 1% 159M 12s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3300K .......... .......... .......... .......... .......... 1% 143M 12s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3350K .......... .......... .......... .......... .......... 2% 135M 12s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3400K .......... .......... .......... .......... .......... 2% 144M 12s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3450K .......... .......... .......... .......... .......... 2% 136M 12s 2024-04-25T03:24:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3500K .......... .......... .......... .......... .......... 2% 33.7M 12s 2024-04-25T03:24:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3550K .......... .......... .......... .......... .......... 2% 2.82M 12s 2024-04-25T03:24:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3600K .......... .......... .......... .......... .......... 2% 121M 12s 2024-04-25T03:24:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3650K .......... .......... .......... .......... .......... 2% 160M 12s 2024-04-25T03:24:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3700K .......... .......... .......... .......... .......... 2% 152M 12s 2024-04-25T03:24:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3750K .......... .......... .......... .......... .......... 2% 131M 12s 2024-04-25T03:24:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3800K .......... .......... .......... .......... .......... 2% 156M 12s 2024-04-25T03:24:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3850K .......... .......... .......... .......... .......... 2% 121M 11s 2024-04-25T03:24:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3900K .......... .......... .......... .......... .......... 2% 146M 11s 2024-04-25T03:24:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 3950K .......... .......... .......... .......... .......... 2% 27.3M 11s 2024-04-25T03:24:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4000K .......... .......... .......... .......... .......... 2% 3.03M 12s 2024-04-25T03:24:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4050K .......... .......... .......... .......... .......... 2% 103M 12s 2024-04-25T03:24:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4100K .......... .......... .......... .......... .......... 2% 157M 11s 2024-04-25T03:24:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4150K .......... .......... .......... .......... .......... 2% 147M 11s 2024-04-25T03:24:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4200K .......... .......... .......... .......... .......... 2% 144M 11s 2024-04-25T03:24:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4250K .......... .......... .......... .......... .......... 2% 165M 11s 2024-04-25T03:24:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4300K .......... .......... .......... .......... .......... 2% 172M 11s 2024-04-25T03:24:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4350K .......... .......... .......... .......... .......... 2% 35.5M 11s 2024-04-25T03:24:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4400K .......... .......... .......... .......... .......... 2% 40.0M 11s 2024-04-25T03:24:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4450K .......... .......... .......... .......... .......... 2% 3.08M 11s 2024-04-25T03:24:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4500K .......... .......... .......... .......... .......... 2% 107M 11s 2024-04-25T03:24:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4550K .......... .......... .......... .......... .......... 2% 92.5M 11s 2024-04-25T03:24:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4600K .......... .......... .......... .......... .......... 2% 128M 11s 2024-04-25T03:24:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4650K .......... .......... .......... .......... .......... 2% 135M 11s 2024-04-25T03:24:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4700K .......... .......... .......... .......... .......... 2% 189M 11s 2024-04-25T03:24:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4750K .......... .......... .......... .......... .......... 2% 152M 11s 2024-04-25T03:24:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4800K .......... .......... .......... .......... .......... 2% 128M 11s 2024-04-25T03:24:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4850K .......... .......... .......... .......... .......... 2% 30.3M 10s 2024-04-25T03:24:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4900K .......... .......... .......... .......... .......... 2% 49.0M 10s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 4950K .......... .......... .......... .......... .......... 2% 3.18M 11s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5000K .......... .......... .......... .......... .......... 2% 82.4M 11s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5050K .......... .......... .......... .......... .......... 3% 88.2M 11s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5100K .......... .......... .......... .......... .......... 3% 88.3M 11s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5150K .......... .......... .......... .......... .......... 3% 73.1M 10s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5200K .......... .......... .......... .......... .......... 3% 91.2M 10s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5250K .......... .......... .......... .......... .......... 3% 165M 10s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5300K .......... .......... .......... .......... .......... 3% 111M 10s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5350K .......... .......... .......... .......... .......... 3% 51.2M 10s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5400K .......... .......... .......... .......... .......... 3% 43.8M 10s 2024-04-25T03:24:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5450K .......... .......... .......... .......... .......... 3% 23.0M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5500K .......... .......... .......... .......... .......... 3% 3.74M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5550K .......... .......... .......... .......... .......... 3% 64.8M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5600K .......... .......... .......... .......... .......... 3% 95.7M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5650K .......... .......... .......... .......... .......... 3% 144M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5700K .......... .......... .......... .......... .......... 3% 112M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5750K .......... .......... .......... .......... .......... 3% 104M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5800K .......... .......... .......... .......... .......... 3% 185M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5850K .......... .......... .......... .......... .......... 3% 99.5M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5900K .......... .......... .......... .......... .......... 3% 48.6M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 5950K .......... .......... .......... .......... .......... 3% 22.5M 10s 2024-04-25T03:24:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6000K .......... .......... .......... .......... .......... 3% 34.1M 10s 2024-04-25T03:24:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6050K .......... .......... .......... .......... .......... 3% 3.76M 10s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6100K .......... .......... .......... .......... .......... 3% 80.1M 10s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6150K .......... .......... .......... .......... .......... 3% 95.4M 10s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6200K .......... .......... .......... .......... .......... 3% 99.9M 10s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6250K .......... .......... .......... .......... .......... 3% 103M 10s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6300K .......... .......... .......... .......... .......... 3% 61.2M 10s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6350K .......... .......... .......... .......... .......... 3% 99.2M 10s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6400K .......... .......... .......... .......... .......... 3% 89.1M 9s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6450K .......... .......... .......... .......... .......... 3% 33.3M 9s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6500K .......... .......... .......... .......... .......... 3% 28.1M 9s 2024-04-25T03:24:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6550K .......... .......... .......... .......... .......... 3% 46.8M 9s 2024-04-25T03:24:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6600K .......... .......... .......... .......... .......... 3% 4.06M 10s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6650K .......... .......... .......... .......... .......... 3% 55.5M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6700K .......... .......... .......... .......... .......... 4% 103M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6750K .......... .......... .......... .......... .......... 4% 75.7M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6800K .......... .......... .......... .......... .......... 4% 99.3M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6850K .......... .......... .......... .......... .......... 4% 92.0M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6900K .......... .......... .......... .......... .......... 4% 34.0M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 6950K .......... .......... .......... .......... .......... 4% 15.8M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7000K .......... .......... .......... .......... .......... 4% 82.8M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7050K .......... .......... .......... .......... .......... 4% 96.3M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7100K .......... .......... .......... .......... .......... 4% 53.8M 9s 2024-04-25T03:24:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7150K .......... .......... .......... .......... .......... 4% 73.4M 9s 2024-04-25T03:24:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7200K .......... .......... .......... .......... .......... 4% 4.81M 9s 2024-04-25T03:24:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7250K .......... .......... .......... .......... .......... 4% 39.1M 9s 2024-04-25T03:24:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7300K .......... .......... .......... .......... .......... 4% 103M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7350K .......... .......... .......... .......... .......... 4% 89.2M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7400K .......... .......... .......... .......... .......... 4% 105M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7450K .......... .......... .......... .......... .......... 4% 101M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7500K .......... .......... .......... .......... .......... 4% 76.6M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7550K .......... .......... .......... .......... .......... 4% 72.6M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7600K .......... .......... .......... .......... .......... 4% 102M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7650K .......... .......... .......... .......... .......... 4% 26.3M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7700K .......... .......... .......... .......... .......... 4% 64.8M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7750K .......... .......... .......... .......... .......... 4% 25.7M 9s 2024-04-25T03:24:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7800K .......... .......... .......... .......... .......... 4% 34.3M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7850K .......... .......... .......... .......... .......... 4% 4.81M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7900K .......... .......... .......... .......... .......... 4% 104M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 7950K .......... .......... .......... .......... .......... 4% 53.3M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8000K .......... .......... .......... .......... .......... 4% 104M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8050K .......... .......... .......... .......... .......... 4% 115M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8100K .......... .......... .......... .......... .......... 4% 156M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8150K .......... .......... .......... .......... .......... 4% 172M 9s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8200K .......... .......... .......... .......... .......... 4% 98.9M 8s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8250K .......... .......... .......... .......... .......... 4% 25.2M 8s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8300K .......... .......... .......... .......... .......... 4% 33.6M 8s 2024-04-25T03:24:16.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8350K .......... .......... .......... .......... .......... 4% 21.8M 8s 2024-04-25T03:24:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8400K .......... .......... .......... .......... .......... 5% 22.3M 8s 2024-04-25T03:24:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8450K .......... .......... .......... .......... .......... 5% 64.9M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8500K .......... .......... .......... .......... .......... 5% 5.65M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8550K .......... .......... .......... .......... .......... 5% 85.1M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8600K .......... .......... .......... .......... .......... 5% 103M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8650K .......... .......... .......... .......... .......... 5% 106M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8700K .......... .......... .......... .......... .......... 5% 109M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8750K .......... .......... .......... .......... .......... 5% 89.0M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8800K .......... .......... .......... .......... .......... 5% 63.4M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8850K .......... .......... .......... .......... .......... 5% 22.1M 8s 2024-04-25T03:24:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8900K .......... .......... .......... .......... .......... 5% 29.5M 8s 2024-04-25T03:24:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 8950K .......... .......... .......... .......... .......... 5% 50.3M 8s 2024-04-25T03:24:16.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9000K .......... .......... .......... .......... .......... 5% 27.4M 8s 2024-04-25T03:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9050K .......... .......... .......... .......... .......... 5% 23.3M 8s 2024-04-25T03:24:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9100K .......... .......... .......... .......... .......... 5% 54.1M 8s 2024-04-25T03:24:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9150K .......... .......... .......... .......... .......... 5% 7.17M 8s 2024-04-25T03:24:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9200K .......... .......... .......... .......... .......... 5% 39.9M 8s 2024-04-25T03:24:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9250K .......... .......... .......... .......... .......... 5% 101M 8s 2024-04-25T03:24:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9300K .......... .......... .......... .......... .......... 5% 116M 8s 2024-04-25T03:24:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9350K .......... .......... .......... .......... .......... 5% 53.8M 8s 2024-04-25T03:24:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9400K .......... .......... .......... .......... .......... 5% 23.5M 8s 2024-04-25T03:24:16.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9450K .......... .......... .......... .......... .......... 5% 60.2M 8s 2024-04-25T03:24:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9500K .......... .......... .......... .......... .......... 5% 24.9M 8s 2024-04-25T03:24:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9550K .......... .......... .......... .......... .......... 5% 23.8M 8s 2024-04-25T03:24:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9600K .......... .......... .......... .......... .......... 5% 26.4M 8s 2024-04-25T03:24:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9650K .......... .......... .......... .......... .......... 5% 29.0M 8s 2024-04-25T03:24:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9700K .......... .......... .......... .......... .......... 5% 54.0M 8s 2024-04-25T03:24:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9750K .......... .......... .......... .......... .......... 5% 25.0M 8s 2024-04-25T03:24:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9800K .......... .......... .......... .......... .......... 5% 24.3M 8s 2024-04-25T03:24:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9850K .......... .......... .......... .......... .......... 5% 31.8M 8s 2024-04-25T03:24:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9900K .......... .......... .......... .......... .......... 5% 23.6M 8s 2024-04-25T03:24:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 9950K .......... .......... .......... .......... .......... 5% 24.7M 8s 2024-04-25T03:24:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10000K .......... .......... .......... .......... .......... 5% 65.0M 8s 2024-04-25T03:24:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10050K .......... .......... .......... .......... .......... 5% 23.6M 8s 2024-04-25T03:24:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10100K .......... .......... .......... .......... .......... 6% 27.5M 8s 2024-04-25T03:24:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10150K .......... .......... .......... .......... .......... 6% 24.6M 8s 2024-04-25T03:24:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10200K .......... .......... .......... .......... .......... 6% 54.0M 8s 2024-04-25T03:24:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10250K .......... .......... .......... .......... .......... 6% 35.2M 8s 2024-04-25T03:24:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10300K .......... .......... .......... .......... .......... 6% 24.5M 8s 2024-04-25T03:24:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10350K .......... .......... .......... .......... .......... 6% 26.7M 8s 2024-04-25T03:24:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10400K .......... .......... .......... .......... .......... 6% 24.9M 8s 2024-04-25T03:24:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10450K .......... .......... .......... .......... .......... 6% 31.5M 8s 2024-04-25T03:24:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10500K .......... .......... .......... .......... .......... 6% 35.5M 8s 2024-04-25T03:24:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10550K .......... .......... .......... .......... .......... 6% 20.4M 8s 2024-04-25T03:24:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10600K .......... .......... .......... .......... .......... 6% 43.7M 8s 2024-04-25T03:24:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10650K .......... .......... .......... .......... .......... 6% 28.0M 8s 2024-04-25T03:24:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10700K .......... .......... .......... .......... .......... 6% 39.2M 8s 2024-04-25T03:24:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10750K .......... .......... .......... .......... .......... 6% 21.1M 8s 2024-04-25T03:24:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10800K .......... .......... .......... .......... .......... 6% 28.2M 8s 2024-04-25T03:24:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10850K .......... .......... .......... .......... .......... 6% 27.8M 8s 2024-04-25T03:24:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10900K .......... .......... .......... .......... .......... 6% 21.8M 8s 2024-04-25T03:24:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 10950K .......... .......... .......... .......... .......... 6% 23.0M 8s 2024-04-25T03:24:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11000K .......... .......... .......... .......... .......... 6% 129M 8s 2024-04-25T03:24:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11050K .......... .......... .......... .......... .......... 6% 32.3M 8s 2024-04-25T03:24:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11100K .......... .......... .......... .......... .......... 6% 23.6M 8s 2024-04-25T03:24:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11150K .......... .......... .......... .......... .......... 6% 24.2M 8s 2024-04-25T03:24:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11200K .......... .......... .......... .......... .......... 6% 22.8M 8s 2024-04-25T03:24:16.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11250K .......... .......... .......... .......... .......... 6% 58.5M 8s 2024-04-25T03:24:16.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11300K .......... .......... .......... .......... .......... 6% 18.3M 8s 2024-04-25T03:24:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11350K .......... .......... .......... .......... .......... 6% 35.8M 8s 2024-04-25T03:24:16.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11400K .......... .......... .......... .......... .......... 6% 34.6M 7s 2024-04-25T03:24:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11450K .......... .......... .......... .......... .......... 6% 24.0M 7s 2024-04-25T03:24:16.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11500K .......... .......... .......... .......... .......... 6% 60.8M 7s 2024-04-25T03:24:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11550K .......... .......... .......... .......... .......... 6% 21.8M 7s 2024-04-25T03:24:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11600K .......... .......... .......... .......... .......... 6% 36.6M 7s 2024-04-25T03:24:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11650K .......... .......... .......... .......... .......... 6% 19.1M 7s 2024-04-25T03:24:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11700K .......... .......... .......... .......... .......... 6% 29.5M 7s 2024-04-25T03:24:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11750K .......... .......... .......... .......... .......... 7% 27.3M 7s 2024-04-25T03:24:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11800K .......... .......... .......... .......... .......... 7% 18.4M 7s 2024-04-25T03:24:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11850K .......... .......... .......... .......... .......... 7% 35.3M 7s 2024-04-25T03:24:17.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11900K .......... .......... .......... .......... .......... 7% 27.0M 7s 2024-04-25T03:24:17.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 11950K .......... .......... .......... .......... .......... 7% 24.0M 7s 2024-04-25T03:24:17.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12000K .......... .......... .......... .......... .......... 7% 50.8M 7s 2024-04-25T03:24:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12050K .......... .......... .......... .......... .......... 7% 23.9M 7s 2024-04-25T03:24:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12100K .......... .......... .......... .......... .......... 7% 34.9M 7s 2024-04-25T03:24:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12150K .......... .......... .......... .......... .......... 7% 23.9M 7s 2024-04-25T03:24:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12200K .......... .......... .......... .......... .......... 7% 9.56M 7s 2024-04-25T03:24:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12250K .......... .......... .......... .......... .......... 7% 66.8M 7s 2024-04-25T03:24:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12300K .......... .......... .......... .......... .......... 7% 30.2M 7s 2024-04-25T03:24:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12350K .......... .......... .......... .......... .......... 7% 20.1M 7s 2024-04-25T03:24:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12400K .......... .......... .......... .......... .......... 7% 28.8M 7s 2024-04-25T03:24:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12450K .......... .......... .......... .......... .......... 7% 21.7M 7s 2024-04-25T03:24:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12500K .......... .......... .......... .......... .......... 7% 107M 7s 2024-04-25T03:24:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12550K .......... .......... .......... .......... .......... 7% 24.2M 7s 2024-04-25T03:24:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12600K .......... .......... .......... .......... .......... 7% 35.8M 7s 2024-04-25T03:24:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12650K .......... .......... .......... .......... .......... 7% 22.2M 7s 2024-04-25T03:24:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12700K .......... .......... .......... .......... .......... 7% 20.6M 7s 2024-04-25T03:24:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12750K .......... .......... .......... .......... .......... 7% 26.9M 7s 2024-04-25T03:24:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12800K .......... .......... .......... .......... .......... 7% 47.9M 7s 2024-04-25T03:24:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12850K .......... .......... .......... .......... .......... 7% 27.1M 7s 2024-04-25T03:24:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12900K .......... .......... .......... .......... .......... 7% 25.4M 7s 2024-04-25T03:24:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 12950K .......... .......... .......... .......... .......... 7% 34.2M 7s 2024-04-25T03:24:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13000K .......... .......... .......... .......... .......... 7% 40.4M 7s 2024-04-25T03:24:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13050K .......... .......... .......... .......... .......... 7% 31.2M 7s 2024-04-25T03:24:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13100K .......... .......... .......... .......... .......... 7% 30.3M 7s 2024-04-25T03:24:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13150K .......... .......... .......... .......... .......... 7% 16.2M 7s 2024-04-25T03:24:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13200K .......... .......... .......... .......... .......... 7% 39.7M 7s 2024-04-25T03:24:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13250K .......... .......... .......... .......... .......... 7% 26.4M 7s 2024-04-25T03:24:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13300K .......... .......... .......... .......... .......... 7% 24.5M 7s 2024-04-25T03:24:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13350K .......... .......... .......... .......... .......... 7% 87.6M 7s 2024-04-25T03:24:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13400K .......... .......... .......... .......... .......... 7% 28.0M 7s 2024-04-25T03:24:17.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13450K .......... .......... .......... .......... .......... 8% 23.7M 7s 2024-04-25T03:24:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13500K .......... .......... .......... .......... .......... 8% 46.8M 7s 2024-04-25T03:24:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13550K .......... .......... .......... .......... .......... 8% 24.1M 7s 2024-04-25T03:24:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13600K .......... .......... .......... .......... .......... 8% 20.9M 7s 2024-04-25T03:24:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13650K .......... .......... .......... .......... .......... 8% 39.0M 7s 2024-04-25T03:24:17.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13700K .......... .......... .......... .......... .......... 8% 23.8M 7s 2024-04-25T03:24:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13750K .......... .......... .......... .......... .......... 8% 31.4M 7s 2024-04-25T03:24:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13800K .......... .......... .......... .......... .......... 8% 27.6M 7s 2024-04-25T03:24:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13850K .......... .......... .......... .......... .......... 8% 12.8M 7s 2024-04-25T03:24:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13900K .......... .......... .......... .......... .......... 8% 39.4M 7s 2024-04-25T03:24:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 13950K .......... .......... .......... .......... .......... 8% 22.4M 7s 2024-04-25T03:24:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14000K .......... .......... .......... .......... .......... 8% 26.5M 7s 2024-04-25T03:24:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14050K .......... .......... .......... .......... .......... 8% 35.8M 7s 2024-04-25T03:24:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14100K .......... .......... .......... .......... .......... 8% 29.3M 7s 2024-04-25T03:24:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14150K .......... .......... .......... .......... .......... 8% 23.5M 7s 2024-04-25T03:24:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14200K .......... .......... .......... .......... .......... 8% 32.3M 7s 2024-04-25T03:24:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14250K .......... .......... .......... .......... .......... 8% 65.7M 7s 2024-04-25T03:24:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14300K .......... .......... .......... .......... .......... 8% 23.8M 7s 2024-04-25T03:24:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14350K .......... .......... .......... .......... .......... 8% 23.2M 7s 2024-04-25T03:24:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14400K .......... .......... .......... .......... .......... 8% 31.4M 7s 2024-04-25T03:24:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14450K .......... .......... .......... .......... .......... 8% 26.9M 7s 2024-04-25T03:24:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14500K .......... .......... .......... .......... .......... 8% 24.3M 7s 2024-04-25T03:24:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14550K .......... .......... .......... .......... .......... 8% 49.3M 7s 2024-04-25T03:24:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14600K .......... .......... .......... .......... .......... 8% 24.4M 7s 2024-04-25T03:24:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14650K .......... .......... .......... .......... .......... 8% 37.7M 7s 2024-04-25T03:24:17.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14700K .......... .......... .......... .......... .......... 8% 24.4M 7s 2024-04-25T03:24:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14750K .......... .......... .......... .......... .......... 8% 20.7M 7s 2024-04-25T03:24:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14800K .......... .......... .......... .......... .......... 8% 59.7M 7s 2024-04-25T03:24:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14850K .......... .......... .......... .......... .......... 8% 22.6M 7s 2024-04-25T03:24:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14900K .......... .......... .......... .......... .......... 8% 37.6M 7s 2024-04-25T03:24:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 14950K .......... .......... .......... .......... .......... 8% 21.3M 7s 2024-04-25T03:24:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15000K .......... .......... .......... .......... .......... 8% 23.0M 7s 2024-04-25T03:24:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15050K .......... .......... .......... .......... .......... 8% 44.0M 7s 2024-04-25T03:24:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15100K .......... .......... .......... .......... .......... 8% 27.7M 7s 2024-04-25T03:24:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15150K .......... .......... .......... .......... .......... 9% 23.9M 7s 2024-04-25T03:24:17.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15200K .......... .......... .......... .......... .......... 9% 27.8M 7s 2024-04-25T03:24:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15250K .......... .......... .......... .......... .......... 9% 9.36M 7s 2024-04-25T03:24:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15300K .......... .......... .......... .......... .......... 9% 81.5M 7s 2024-04-25T03:24:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15350K .......... .......... .......... .......... .......... 9% 25.9M 7s 2024-04-25T03:24:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15400K .......... .......... .......... .......... .......... 9% 39.4M 7s 2024-04-25T03:24:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15450K .......... .......... .......... .......... .......... 9% 32.3M 7s 2024-04-25T03:24:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15500K .......... .......... .......... .......... .......... 9% 17.5M 7s 2024-04-25T03:24:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15550K .......... .......... .......... .......... .......... 9% 23.6M 7s 2024-04-25T03:24:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15600K .......... .......... .......... .......... .......... 9% 47.4M 7s 2024-04-25T03:24:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15650K .......... .......... .......... .......... .......... 9% 24.7M 7s 2024-04-25T03:24:17.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15700K .......... .......... .......... .......... .......... 9% 38.1M 7s 2024-04-25T03:24:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15750K .......... .......... .......... .......... .......... 9% 23.9M 7s 2024-04-25T03:24:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15800K .......... .......... .......... .......... .......... 9% 48.3M 7s 2024-04-25T03:24:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15850K .......... .......... .......... .......... .......... 9% 22.3M 7s 2024-04-25T03:24:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15900K .......... .......... .......... .......... .......... 9% 52.2M 7s 2024-04-25T03:24:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 15950K .......... .......... .......... .......... .......... 9% 19.1M 7s 2024-04-25T03:24:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16000K .......... .......... .......... .......... .......... 9% 25.2M 7s 2024-04-25T03:24:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16050K .......... .......... .......... .......... .......... 9% 20.2M 7s 2024-04-25T03:24:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16100K .......... .......... .......... .......... .......... 9% 50.0M 7s 2024-04-25T03:24:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16150K .......... .......... .......... .......... .......... 9% 27.1M 7s 2024-04-25T03:24:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16200K .......... .......... .......... .......... .......... 9% 41.8M 7s 2024-04-25T03:24:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16250K .......... .......... .......... .......... .......... 9% 25.6M 7s 2024-04-25T03:24:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16300K .......... .......... .......... .......... .......... 9% 38.9M 7s 2024-04-25T03:24:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16350K .......... .......... .......... .......... .......... 9% 19.5M 7s 2024-04-25T03:24:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16400K .......... .......... .......... .......... .......... 9% 35.7M 7s 2024-04-25T03:24:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16450K .......... .......... .......... .......... .......... 9% 32.7M 7s 2024-04-25T03:24:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16500K .......... .......... .......... .......... .......... 9% 16.3M 7s 2024-04-25T03:24:17.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16550K .......... .......... .......... .......... .......... 9% 46.5M 7s 2024-04-25T03:24:17.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16600K .......... .......... .......... .......... .......... 9% 53.5M 7s 2024-04-25T03:24:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16650K .......... .......... .......... .......... .......... 9% 23.7M 7s 2024-04-25T03:24:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16700K .......... .......... .......... .......... .......... 9% 40.4M 7s 2024-04-25T03:24:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16750K .......... .......... .......... .......... .......... 9% 23.5M 7s 2024-04-25T03:24:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16800K .......... .......... .......... .......... .......... 9% 28.4M 7s 2024-04-25T03:24:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16850K .......... .......... .......... .......... .......... 10% 37.2M 7s 2024-04-25T03:24:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16900K .......... .......... .......... .......... .......... 10% 23.8M 7s 2024-04-25T03:24:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 16950K .......... .......... .......... .......... .......... 10% 47.6M 7s 2024-04-25T03:24:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17000K .......... .......... .......... .......... .......... 10% 19.2M 7s 2024-04-25T03:24:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17050K .......... .......... .......... .......... .......... 10% 37.0M 7s 2024-04-25T03:24:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17100K .......... .......... .......... .......... .......... 10% 28.9M 7s 2024-04-25T03:24:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17150K .......... .......... .......... .......... .......... 10% 25.8M 7s 2024-04-25T03:24:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17200K .......... .......... .......... .......... .......... 10% 23.7M 7s 2024-04-25T03:24:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17250K .......... .......... .......... .......... .......... 10% 40.3M 7s 2024-04-25T03:24:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17300K .......... .......... .......... .......... .......... 10% 22.6M 7s 2024-04-25T03:24:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17350K .......... .......... .......... .......... .......... 10% 46.1M 7s 2024-04-25T03:24:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17400K .......... .......... .......... .......... .......... 10% 22.6M 7s 2024-04-25T03:24:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17450K .......... .......... .......... .......... .......... 10% 41.0M 7s 2024-04-25T03:24:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17500K .......... .......... .......... .......... .......... 10% 24.9M 7s 2024-04-25T03:24:17.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17550K .......... .......... .......... .......... .......... 10% 24.9M 7s 2024-04-25T03:24:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17600K .......... .......... .......... .......... .......... 10% 44.9M 7s 2024-04-25T03:24:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17650K .......... .......... .......... .......... .......... 10% 24.8M 7s 2024-04-25T03:24:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17700K .......... .......... .......... .......... .......... 10% 29.4M 7s 2024-04-25T03:24:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17750K .......... .......... .......... .......... .......... 10% 25.9M 7s 2024-04-25T03:24:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17800K .......... .......... .......... .......... .......... 10% 23.2M 7s 2024-04-25T03:24:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17850K .......... .......... .......... .......... .......... 10% 45.7M 7s 2024-04-25T03:24:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17900K .......... .......... .......... .......... .......... 10% 28.7M 7s 2024-04-25T03:24:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 17950K .......... .......... .......... .......... .......... 10% 22.5M 7s 2024-04-25T03:24:17.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18000K .......... .......... .......... .......... .......... 10% 33.9M 7s 2024-04-25T03:24:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18050K .......... .......... .......... .......... .......... 10% 27.1M 6s 2024-04-25T03:24:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18100K .......... .......... .......... .......... .......... 10% 51.7M 6s 2024-04-25T03:24:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18150K .......... .......... .......... .......... .......... 10% 23.5M 6s 2024-04-25T03:24:17.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18200K .......... .......... .......... .......... .......... 10% 34.2M 6s 2024-04-25T03:24:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18250K .......... .......... .......... .......... .......... 10% 24.0M 6s 2024-04-25T03:24:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18300K .......... .......... .......... .......... .......... 10% 21.6M 6s 2024-04-25T03:24:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18350K .......... .......... .......... .......... .......... 10% 32.6M 6s 2024-04-25T03:24:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18400K .......... .......... .......... .......... .......... 10% 41.4M 6s 2024-04-25T03:24:17.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18450K .......... .......... .......... .......... .......... 10% 23.8M 6s 2024-04-25T03:24:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18500K .......... .......... .......... .......... .......... 11% 34.3M 6s 2024-04-25T03:24:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18550K .......... .......... .......... .......... .......... 11% 22.6M 6s 2024-04-25T03:24:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18600K .......... .......... .......... .......... .......... 11% 64.4M 6s 2024-04-25T03:24:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18650K .......... .......... .......... .......... .......... 11% 24.3M 6s 2024-04-25T03:24:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18700K .......... .......... .......... .......... .......... 11% 29.5M 6s 2024-04-25T03:24:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18750K .......... .......... .......... .......... .......... 11% 23.3M 6s 2024-04-25T03:24:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18800K .......... .......... .......... .......... .......... 11% 26.6M 6s 2024-04-25T03:24:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18850K .......... .......... .......... .......... .......... 11% 22.8M 6s 2024-04-25T03:24:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18900K .......... .......... .......... .......... .......... 11% 64.6M 6s 2024-04-25T03:24:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 18950K .......... .......... .......... .......... .......... 11% 35.5M 6s 2024-04-25T03:24:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19000K .......... .......... .......... .......... .......... 11% 21.7M 6s 2024-04-25T03:24:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19050K .......... .......... .......... .......... .......... 11% 24.7M 6s 2024-04-25T03:24:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19100K .......... .......... .......... .......... .......... 11% 64.0M 6s 2024-04-25T03:24:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19150K .......... .......... .......... .......... .......... 11% 21.1M 6s 2024-04-25T03:24:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19200K .......... .......... .......... .......... .......... 11% 29.4M 6s 2024-04-25T03:24:17.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19250K .......... .......... .......... .......... .......... 11% 30.8M 6s 2024-04-25T03:24:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19300K .......... .......... .......... .......... .......... 11% 25.7M 6s 2024-04-25T03:24:17.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19350K .......... .......... .......... .......... .......... 11% 56.5M 6s 2024-04-25T03:24:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19400K .......... .......... .......... .......... .......... 11% 24.3M 6s 2024-04-25T03:24:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19450K .......... .......... .......... .......... .......... 11% 33.5M 6s 2024-04-25T03:24:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19500K .......... .......... .......... .......... .......... 11% 25.3M 6s 2024-04-25T03:24:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19550K .......... .......... .......... .......... .......... 11% 20.0M 6s 2024-04-25T03:24:17.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19600K .......... .......... .......... .......... .......... 11% 29.7M 6s 2024-04-25T03:24:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19650K .......... .......... .......... .......... .......... 11% 41.3M 6s 2024-04-25T03:24:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19700K .......... .......... .......... .......... .......... 11% 28.2M 6s 2024-04-25T03:24:17.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19750K .......... .......... .......... .......... .......... 11% 32.1M 6s 2024-04-25T03:24:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19800K .......... .......... .......... .......... .......... 11% 24.2M 6s 2024-04-25T03:24:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19850K .......... .......... .......... .......... .......... 11% 44.7M 6s 2024-04-25T03:24:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19900K .......... .......... .......... .......... .......... 11% 16.2M 6s 2024-04-25T03:24:17.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 19950K .......... .......... .......... .......... .......... 11% 24.0M 6s 2024-04-25T03:24:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20000K .......... .......... .......... .......... .......... 11% 31.3M 6s 2024-04-25T03:24:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20050K .......... .......... .......... .......... .......... 11% 23.9M 6s 2024-04-25T03:24:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20100K .......... .......... .......... .......... .......... 11% 26.8M 6s 2024-04-25T03:24:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20150K .......... .......... .......... .......... .......... 11% 32.8M 6s 2024-04-25T03:24:17.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20200K .......... .......... .......... .......... .......... 12% 30.1M 6s 2024-04-25T03:24:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20250K .......... .......... .......... .......... .......... 12% 24.2M 6s 2024-04-25T03:24:17.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20300K .......... .......... .......... .......... .......... 12% 17.7M 6s 2024-04-25T03:24:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20350K .......... .......... .......... .......... .......... 12% 32.7M 6s 2024-04-25T03:24:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20400K .......... .......... .......... .......... .......... 12% 65.1M 6s 2024-04-25T03:24:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20450K .......... .......... .......... .......... .......... 12% 25.5M 6s 2024-04-25T03:24:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20500K .......... .......... .......... .......... .......... 12% 34.1M 6s 2024-04-25T03:24:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20550K .......... .......... .......... .......... .......... 12% 22.8M 6s 2024-04-25T03:24:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20600K .......... .......... .......... .......... .......... 12% 23.5M 6s 2024-04-25T03:24:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20650K .......... .......... .......... .......... .......... 12% 57.4M 6s 2024-04-25T03:24:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20700K .......... .......... .......... .......... .......... 12% 19.3M 6s 2024-04-25T03:24:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20750K .......... .......... .......... .......... .......... 12% 30.9M 6s 2024-04-25T03:24:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20800K .......... .......... .......... .......... .......... 12% 34.8M 6s 2024-04-25T03:24:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20850K .......... .......... .......... .......... .......... 12% 25.7M 6s 2024-04-25T03:24:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20900K .......... .......... .......... .......... .......... 12% 55.9M 6s 2024-04-25T03:24:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 20950K .......... .......... .......... .......... .......... 12% 22.1M 6s 2024-04-25T03:24:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21000K .......... .......... .......... .......... .......... 12% 39.1M 6s 2024-04-25T03:24:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21050K .......... .......... .......... .......... .......... 12% 23.6M 6s 2024-04-25T03:24:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21100K .......... .......... .......... .......... .......... 12% 20.7M 6s 2024-04-25T03:24:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21150K .......... .......... .......... .......... .......... 12% 28.3M 6s 2024-04-25T03:24:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21200K .......... .......... .......... .......... .......... 12% 65.5M 6s 2024-04-25T03:24:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21250K .......... .......... .......... .......... .......... 12% 33.8M 6s 2024-04-25T03:24:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21300K .......... .......... .......... .......... .......... 12% 24.1M 6s 2024-04-25T03:24:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21350K .......... .......... .......... .......... .......... 12% 24.0M 6s 2024-04-25T03:24:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21400K .......... .......... .......... .......... .......... 12% 39.4M 6s 2024-04-25T03:24:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21450K .......... .......... .......... .......... .......... 12% 18.0M 6s 2024-04-25T03:24:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21500K .......... .......... .......... .......... .......... 12% 29.9M 6s 2024-04-25T03:24:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21550K .......... .......... .......... .......... .......... 12% 35.6M 6s 2024-04-25T03:24:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21600K .......... .......... .......... .......... .......... 12% 31.8M 6s 2024-04-25T03:24:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21650K .......... .......... .......... .......... .......... 12% 22.3M 6s 2024-04-25T03:24:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21700K .......... .......... .......... .......... .......... 12% 120M 6s 2024-04-25T03:24:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21750K .......... .......... .......... .......... .......... 12% 22.8M 6s 2024-04-25T03:24:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21800K .......... .......... .......... .......... .......... 12% 24.7M 6s 2024-04-25T03:24:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21850K .......... .......... .......... .......... .......... 12% 22.7M 6s 2024-04-25T03:24:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21900K .......... .......... .......... .......... .......... 13% 42.7M 6s 2024-04-25T03:24:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 21950K .......... .......... .......... .......... .......... 13% 22.4M 6s 2024-04-25T03:24:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22000K .......... .......... .......... .......... .......... 13% 29.1M 6s 2024-04-25T03:24:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22050K .......... .......... .......... .......... .......... 13% 25.0M 6s 2024-04-25T03:24:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22100K .......... .......... .......... .......... .......... 13% 25.6M 6s 2024-04-25T03:24:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22150K .......... .......... .......... .......... .......... 13% 16.1M 6s 2024-04-25T03:24:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22200K .......... .......... .......... .......... .......... 13% 19.1M 6s 2024-04-25T03:24:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22250K .......... .......... .......... .......... .......... 13% 21.3M 6s 2024-04-25T03:24:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22300K .......... .......... .......... .......... .......... 13% 19.3M 6s 2024-04-25T03:24:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22350K .......... .......... .......... .......... .......... 13% 26.0M 6s 2024-04-25T03:24:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22400K .......... .......... .......... .......... .......... 13% 23.6M 6s 2024-04-25T03:24:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22450K .......... .......... .......... .......... .......... 13% 51.8M 6s 2024-04-25T03:24:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22500K .......... .......... .......... .......... .......... 13% 25.6M 6s 2024-04-25T03:24:17.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22550K .......... .......... .......... .......... .......... 13% 35.8M 6s 2024-04-25T03:24:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22600K .......... .......... .......... .......... .......... 13% 22.7M 6s 2024-04-25T03:24:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22650K .......... .......... .......... .......... .......... 13% 59.0M 6s 2024-04-25T03:24:17.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22700K .......... .......... .......... .......... .......... 13% 29.5M 6s 2024-04-25T03:24:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22750K .......... .......... .......... .......... .......... 13% 8.06M 6s 2024-04-25T03:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22800K .......... .......... .......... .......... .......... 13% 77.2M 6s 2024-04-25T03:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22850K .......... .......... .......... .......... .......... 13% 105M 6s 2024-04-25T03:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22900K .......... .......... .......... .......... .......... 13% 61.7M 6s 2024-04-25T03:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 22950K .......... .......... .......... .......... .......... 13% 62.9M 6s 2024-04-25T03:24:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23000K .......... .......... .......... .......... .......... 13% 21.4M 6s 2024-04-25T03:24:17.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23050K .......... .......... .......... .......... .......... 13% 35.1M 6s 2024-04-25T03:24:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23100K .......... .......... .......... .......... .......... 13% 27.9M 6s 2024-04-25T03:24:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23150K .......... .......... .......... .......... .......... 13% 23.7M 6s 2024-04-25T03:24:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23200K .......... .......... .......... .......... .......... 13% 53.8M 6s 2024-04-25T03:24:17.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23250K .......... .......... .......... .......... .......... 13% 25.7M 6s 2024-04-25T03:24:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23300K .......... .......... .......... .......... .......... 13% 16.5M 6s 2024-04-25T03:24:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23350K .......... .......... .......... .......... .......... 13% 31.9M 6s 2024-04-25T03:24:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23400K .......... .......... .......... .......... .......... 13% 42.8M 6s 2024-04-25T03:24:17.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23450K .......... .......... .......... .......... .......... 13% 48.8M 6s 2024-04-25T03:24:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23500K .......... .......... .......... .......... .......... 13% 25.8M 6s 2024-04-25T03:24:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23550K .......... .......... .......... .......... .......... 14% 22.5M 6s 2024-04-25T03:24:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23600K .......... .......... .......... .......... .......... 14% 34.5M 6s 2024-04-25T03:24:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23650K .......... .......... .......... .......... .......... 14% 22.8M 6s 2024-04-25T03:24:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23700K .......... .......... .......... .......... .......... 14% 40.4M 6s 2024-04-25T03:24:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23750K .......... .......... .......... .......... .......... 14% 22.2M 6s 2024-04-25T03:24:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23800K .......... .......... .......... .......... .......... 14% 46.7M 6s 2024-04-25T03:24:17.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23850K .......... .......... .......... .......... .......... 14% 24.0M 6s 2024-04-25T03:24:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23900K .......... .......... .......... .......... .......... 14% 52.6M 6s 2024-04-25T03:24:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 23950K .......... .......... .......... .......... .......... 14% 15.6M 6s 2024-04-25T03:24:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24000K .......... .......... .......... .......... .......... 14% 40.8M 6s 2024-04-25T03:24:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24050K .......... .......... .......... .......... .......... 14% 20.6M 6s 2024-04-25T03:24:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24100K .......... .......... .......... .......... .......... 14% 37.1M 6s 2024-04-25T03:24:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24150K .......... .......... .......... .......... .......... 14% 32.9M 6s 2024-04-25T03:24:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24200K .......... .......... .......... .......... .......... 14% 35.9M 6s 2024-04-25T03:24:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24250K .......... .......... .......... .......... .......... 14% 27.1M 6s 2024-04-25T03:24:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24300K .......... .......... .......... .......... .......... 14% 43.0M 6s 2024-04-25T03:24:17.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24350K .......... .......... .......... .......... .......... 14% 22.1M 6s 2024-04-25T03:24:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24400K .......... .......... .......... .......... .......... 14% 25.1M 6s 2024-04-25T03:24:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24450K .......... .......... .......... .......... .......... 14% 44.4M 6s 2024-04-25T03:24:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24500K .......... .......... .......... .......... .......... 14% 22.7M 6s 2024-04-25T03:24:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24550K .......... .......... .......... .......... .......... 14% 25.8M 6s 2024-04-25T03:24:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24600K .......... .......... .......... .......... .......... 14% 55.2M 6s 2024-04-25T03:24:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24650K .......... .......... .......... .......... .......... 14% 24.3M 6s 2024-04-25T03:24:17.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24700K .......... .......... .......... .......... .......... 14% 42.2M 6s 2024-04-25T03:24:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24750K .......... .......... .......... .......... .......... 14% 20.9M 6s 2024-04-25T03:24:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24800K .......... .......... .......... .......... .......... 14% 25.6M 6s 2024-04-25T03:24:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24850K .......... .......... .......... .......... .......... 14% 40.7M 6s 2024-04-25T03:24:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24900K .......... .......... .......... .......... .......... 14% 18.7M 6s 2024-04-25T03:24:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 24950K .......... .......... .......... .......... .......... 14% 88.6M 6s 2024-04-25T03:24:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25000K .......... .......... .......... .......... .......... 14% 22.7M 6s 2024-04-25T03:24:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25050K .......... .......... .......... .......... .......... 14% 43.4M 6s 2024-04-25T03:24:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25100K .......... .......... .......... .......... .......... 14% 20.3M 6s 2024-04-25T03:24:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25150K .......... .......... .......... .......... .......... 14% 33.7M 6s 2024-04-25T03:24:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25200K .......... .......... .......... .......... .......... 14% 20.6M 6s 2024-04-25T03:24:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25250K .......... .......... .......... .......... .......... 15% 49.7M 6s 2024-04-25T03:24:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25300K .......... .......... .......... .......... .......... 15% 22.0M 6s 2024-04-25T03:24:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25350K .......... .......... .......... .......... .......... 15% 35.3M 6s 2024-04-25T03:24:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25400K .......... .......... .......... .......... .......... 15% 27.8M 6s 2024-04-25T03:24:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25450K .......... .......... .......... .......... .......... 15% 51.5M 6s 2024-04-25T03:24:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25500K .......... .......... .......... .......... .......... 15% 24.9M 6s 2024-04-25T03:24:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25550K .......... .......... .......... .......... .......... 15% 23.9M 6s 2024-04-25T03:24:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25600K .......... .......... .......... .......... .......... 15% 31.4M 6s 2024-04-25T03:24:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25650K .......... .......... .......... .......... .......... 15% 24.8M 6s 2024-04-25T03:24:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25700K .......... .......... .......... .......... .......... 15% 23.2M 6s 2024-04-25T03:24:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25750K .......... .......... .......... .......... .......... 15% 66.2M 6s 2024-04-25T03:24:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25800K .......... .......... .......... .......... .......... 15% 33.6M 6s 2024-04-25T03:24:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25850K .......... .......... .......... .......... .......... 15% 24.1M 6s 2024-04-25T03:24:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25900K .......... .......... .......... .......... .......... 15% 30.4M 6s 2024-04-25T03:24:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 25950K .......... .......... .......... .......... .......... 15% 30.7M 6s 2024-04-25T03:24:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26000K .......... .......... .......... .......... .......... 15% 30.2M 6s 2024-04-25T03:24:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26050K .......... .......... .......... .......... .......... 15% 21.9M 6s 2024-04-25T03:24:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26100K .......... .......... .......... .......... .......... 15% 30.7M 6s 2024-04-25T03:24:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26150K .......... .......... .......... .......... .......... 15% 22.2M 6s 2024-04-25T03:24:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26200K .......... .......... .......... .......... .......... 15% 42.9M 6s 2024-04-25T03:24:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26250K .......... .......... .......... .......... .......... 15% 46.3M 6s 2024-04-25T03:24:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26300K .......... .......... .......... .......... .......... 15% 27.0M 6s 2024-04-25T03:24:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26350K .......... .......... .......... .......... .......... 15% 18.1M 6s 2024-04-25T03:24:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26400K .......... .......... .......... .......... .......... 15% 37.6M 6s 2024-04-25T03:24:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26450K .......... .......... .......... .......... .......... 15% 25.9M 6s 2024-04-25T03:24:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26500K .......... .......... .......... .......... .......... 15% 45.5M 6s 2024-04-25T03:24:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26550K .......... .......... .......... .......... .......... 15% 24.4M 6s 2024-04-25T03:24:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26600K .......... .......... .......... .......... .......... 15% 34.0M 6s 2024-04-25T03:24:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26650K .......... .......... .......... .......... .......... 15% 25.4M 6s 2024-04-25T03:24:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26700K .......... .......... .......... .......... .......... 15% 47.3M 6s 2024-04-25T03:24:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26750K .......... .......... .......... .......... .......... 15% 15.5M 6s 2024-04-25T03:24:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26800K .......... .......... .......... .......... .......... 15% 63.4M 6s 2024-04-25T03:24:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26850K .......... .......... .......... .......... .......... 15% 20.3M 6s 2024-04-25T03:24:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26900K .......... .......... .......... .......... .......... 15% 38.1M 6s 2024-04-25T03:24:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 26950K .......... .......... .......... .......... .......... 16% 24.6M 6s 2024-04-25T03:24:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27000K .......... .......... .......... .......... .......... 16% 49.1M 6s 2024-04-25T03:24:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27050K .......... .......... .......... .......... .......... 16% 22.4M 6s 2024-04-25T03:24:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27100K .......... .......... .......... .......... .......... 16% 42.0M 6s 2024-04-25T03:24:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27150K .......... .......... .......... .......... .......... 16% 24.4M 6s 2024-04-25T03:24:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27200K .......... .......... .......... .......... .......... 16% 21.8M 6s 2024-04-25T03:24:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27250K .......... .......... .......... .......... .......... 16% 80.2M 6s 2024-04-25T03:24:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27300K .......... .......... .......... .......... .......... 16% 22.6M 6s 2024-04-25T03:24:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27350K .......... .......... .......... .......... .......... 16% 31.6M 6s 2024-04-25T03:24:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27400K .......... .......... .......... .......... .......... 16% 41.0M 6s 2024-04-25T03:24:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27450K .......... .......... .......... .......... .......... 16% 18.6M 6s 2024-04-25T03:24:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27500K .......... .......... .......... .......... .......... 16% 57.0M 6s 2024-04-25T03:24:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27550K .......... .......... .......... .......... .......... 16% 22.8M 6s 2024-04-25T03:24:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27600K .......... .......... .......... .......... .......... 16% 24.5M 6s 2024-04-25T03:24:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27650K .......... .......... .......... .......... .......... 16% 35.7M 6s 2024-04-25T03:24:17.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27700K .......... .......... .......... .......... .......... 16% 30.2M 6s 2024-04-25T03:24:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27750K .......... .......... .......... .......... .......... 16% 28.4M 6s 2024-04-25T03:24:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27800K .......... .......... .......... .......... .......... 16% 29.6M 6s 2024-04-25T03:24:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27850K .......... .......... .......... .......... .......... 16% 29.9M 6s 2024-04-25T03:24:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27900K .......... .......... .......... .......... .......... 16% 22.6M 6s 2024-04-25T03:24:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 27950K .......... .......... .......... .......... .......... 16% 27.9M 6s 2024-04-25T03:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28000K .......... .......... .......... .......... .......... 16% 23.2M 6s 2024-04-25T03:24:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28050K .......... .......... .......... .......... .......... 16% 39.2M 6s 2024-04-25T03:24:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28100K .......... .......... .......... .......... .......... 16% 23.5M 6s 2024-04-25T03:24:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28150K .......... .......... .......... .......... .......... 16% 36.2M 6s 2024-04-25T03:24:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28200K .......... .......... .......... .......... .......... 16% 24.7M 6s 2024-04-25T03:24:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28250K .......... .......... .......... .......... .......... 16% 38.1M 6s 2024-04-25T03:24:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28300K .......... .......... .......... .......... .......... 16% 28.9M 6s 2024-04-25T03:24:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28350K .......... .......... .......... .......... .......... 16% 10.8M 6s 2024-04-25T03:24:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28400K .......... .......... .......... .......... .......... 16% 39.1M 6s 2024-04-25T03:24:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28450K .......... .......... .......... .......... .......... 16% 20.9M 6s 2024-04-25T03:24:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28500K .......... .......... .......... .......... .......... 16% 39.3M 6s 2024-04-25T03:24:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28550K .......... .......... .......... .......... .......... 16% 37.8M 6s 2024-04-25T03:24:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28600K .......... .......... .......... .......... .......... 16% 10.3M 6s 2024-04-25T03:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28650K .......... .......... .......... .......... .......... 17% 19.0M 6s 2024-04-25T03:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28700K .......... .......... .......... .......... .......... 17% 34.4M 6s 2024-04-25T03:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28750K .......... .......... .......... .......... .......... 17% 22.5M 6s 2024-04-25T03:24:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28800K .......... .......... .......... .......... .......... 17% 45.0M 6s 2024-04-25T03:24:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28850K .......... .......... .......... .......... .......... 17% 25.9M 6s 2024-04-25T03:24:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28900K .......... .......... .......... .......... .......... 17% 32.5M 6s 2024-04-25T03:24:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 28950K .......... .......... .......... .......... .......... 17% 24.6M 6s 2024-04-25T03:24:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29000K .......... .......... .......... .......... .......... 17% 26.5M 6s 2024-04-25T03:24:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29050K .......... .......... .......... .......... .......... 17% 98.1M 6s 2024-04-25T03:24:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29100K .......... .......... .......... .......... .......... 17% 26.8M 6s 2024-04-25T03:24:17.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29150K .......... .......... .......... .......... .......... 17% 22.6M 6s 2024-04-25T03:24:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29200K .......... .......... .......... .......... .......... 17% 25.6M 6s 2024-04-25T03:24:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29250K .......... .......... .......... .......... .......... 17% 25.6M 6s 2024-04-25T03:24:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29300K .......... .......... .......... .......... .......... 17% 40.5M 6s 2024-04-25T03:24:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29350K .......... .......... .......... .......... .......... 17% 22.6M 6s 2024-04-25T03:24:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29400K .......... .......... .......... .......... .......... 17% 34.7M 6s 2024-04-25T03:24:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29450K .......... .......... .......... .......... .......... 17% 30.0M 6s 2024-04-25T03:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29500K .......... .......... .......... .......... .......... 17% 42.7M 6s 2024-04-25T03:24:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29550K .......... .......... .......... .......... .......... 17% 26.4M 6s 2024-04-25T03:24:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29600K .......... .......... .......... .......... .......... 17% 24.8M 6s 2024-04-25T03:24:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29650K .......... .......... .......... .......... .......... 17% 33.2M 6s 2024-04-25T03:24:17.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29700K .......... .......... .......... .......... .......... 17% 21.6M 6s 2024-04-25T03:24:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29750K .......... .......... .......... .......... .......... 17% 18.1M 6s 2024-04-25T03:24:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29800K .......... .......... .......... .......... .......... 17% 40.3M 6s 2024-04-25T03:24:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29850K .......... .......... .......... .......... .......... 17% 43.3M 6s 2024-04-25T03:24:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29900K .......... .......... .......... .......... .......... 17% 35.6M 6s 2024-04-25T03:24:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 29950K .......... .......... .......... .......... .......... 17% 31.6M 6s 2024-04-25T03:24:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30000K .......... .......... .......... .......... .......... 17% 27.7M 6s 2024-04-25T03:24:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30050K .......... .......... .......... .......... .......... 17% 33.3M 6s 2024-04-25T03:24:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30100K .......... .......... .......... .......... .......... 17% 22.8M 6s 2024-04-25T03:24:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30150K .......... .......... .......... .......... .......... 17% 33.6M 6s 2024-04-25T03:24:17.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30200K .......... .......... .......... .......... .......... 17% 23.7M 6s 2024-04-25T03:24:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30250K .......... .......... .......... .......... .......... 17% 22.8M 6s 2024-04-25T03:24:17.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30300K .......... .......... .......... .......... .......... 18% 59.7M 5s 2024-04-25T03:24:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30350K .......... .......... .......... .......... .......... 18% 24.4M 5s 2024-04-25T03:24:17.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30400K .......... .......... .......... .......... .......... 18% 21.2M 5s 2024-04-25T03:24:17.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30450K .......... .......... .......... .......... .......... 18% 30.2M 5s 2024-04-25T03:24:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30500K .......... .......... .......... .......... .......... 18% 27.2M 5s 2024-04-25T03:24:17.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30550K .......... .......... .......... .......... .......... 18% 31.7M 5s 2024-04-25T03:24:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30600K .......... .......... .......... .......... .......... 18% 15.7M 5s 2024-04-25T03:24:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30650K .......... .......... .......... .......... .......... 18% 42.0M 5s 2024-04-25T03:24:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30700K .......... .......... .......... .......... .......... 18% 23.4M 5s 2024-04-25T03:24:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30750K .......... .......... .......... .......... .......... 18% 22.1M 5s 2024-04-25T03:24:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30800K .......... .......... .......... .......... .......... 18% 26.8M 5s 2024-04-25T03:24:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30850K .......... .......... .......... .......... .......... 18% 40.4M 5s 2024-04-25T03:24:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30900K .......... .......... .......... .......... .......... 18% 21.4M 5s 2024-04-25T03:24:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 30950K .......... .......... .......... .......... .......... 18% 30.8M 5s 2024-04-25T03:24:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31000K .......... .......... .......... .......... .......... 18% 38.2M 5s 2024-04-25T03:24:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31050K .......... .......... .......... .......... .......... 18% 42.8M 5s 2024-04-25T03:24:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31100K .......... .......... .......... .......... .......... 18% 23.1M 5s 2024-04-25T03:24:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31150K .......... .......... .......... .......... .......... 18% 24.0M 5s 2024-04-25T03:24:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31200K .......... .......... .......... .......... .......... 18% 41.9M 5s 2024-04-25T03:24:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31250K .......... .......... .......... .......... .......... 18% 24.0M 5s 2024-04-25T03:24:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31300K .......... .......... .......... .......... .......... 18% 18.0M 5s 2024-04-25T03:24:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31350K .......... .......... .......... .......... .......... 18% 52.1M 5s 2024-04-25T03:24:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31400K .......... .......... .......... .......... .......... 18% 25.8M 5s 2024-04-25T03:24:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31450K .......... .......... .......... .......... .......... 18% 52.2M 5s 2024-04-25T03:24:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31500K .......... .......... .......... .......... .......... 18% 23.3M 5s 2024-04-25T03:24:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31550K .......... .......... .......... .......... .......... 18% 25.9M 5s 2024-04-25T03:24:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31600K .......... .......... .......... .......... .......... 18% 46.4M 5s 2024-04-25T03:24:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31650K .......... .......... .......... .......... .......... 18% 23.8M 5s 2024-04-25T03:24:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31700K .......... .......... .......... .......... .......... 18% 40.7M 5s 2024-04-25T03:24:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31750K .......... .......... .......... .......... .......... 18% 15.9M 5s 2024-04-25T03:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31800K .......... .......... .......... .......... .......... 18% 85.4M 5s 2024-04-25T03:24:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31850K .......... .......... .......... .......... .......... 18% 26.4M 5s 2024-04-25T03:24:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31900K .......... .......... .......... .......... .......... 18% 31.9M 5s 2024-04-25T03:24:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 31950K .......... .......... .......... .......... .......... 18% 20.9M 5s 2024-04-25T03:24:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32000K .......... .......... .......... .......... .......... 19% 21.4M 5s 2024-04-25T03:24:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32050K .......... .......... .......... .......... .......... 19% 37.6M 5s 2024-04-25T03:24:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32100K .......... .......... .......... .......... .......... 19% 50.1M 5s 2024-04-25T03:24:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32150K .......... .......... .......... .......... .......... 19% 20.7M 5s 2024-04-25T03:24:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32200K .......... .......... .......... .......... .......... 19% 30.4M 5s 2024-04-25T03:24:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32250K .......... .......... .......... .......... .......... 19% 31.4M 5s 2024-04-25T03:24:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32300K .......... .......... .......... .......... .......... 19% 54.3M 5s 2024-04-25T03:24:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32350K .......... .......... .......... .......... .......... 19% 24.2M 5s 2024-04-25T03:24:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32400K .......... .......... .......... .......... .......... 19% 24.0M 5s 2024-04-25T03:24:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32450K .......... .......... .......... .......... .......... 19% 33.5M 5s 2024-04-25T03:24:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32500K .......... .......... .......... .......... .......... 19% 24.0M 5s 2024-04-25T03:24:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32550K .......... .......... .......... .......... .......... 19% 25.4M 5s 2024-04-25T03:24:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32600K .......... .......... .......... .......... .......... 19% 45.4M 5s 2024-04-25T03:24:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32650K .......... .......... .......... .......... .......... 19% 30.4M 5s 2024-04-25T03:24:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32700K .......... .......... .......... .......... .......... 19% 24.4M 5s 2024-04-25T03:24:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32750K .......... .......... .......... .......... .......... 19% 31.3M 5s 2024-04-25T03:24:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32800K .......... .......... .......... .......... .......... 19% 21.4M 5s 2024-04-25T03:24:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32850K .......... .......... .......... .......... .......... 19% 41.8M 5s 2024-04-25T03:24:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32900K .......... .......... .......... .......... .......... 19% 26.8M 5s 2024-04-25T03:24:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 32950K .......... .......... .......... .......... .......... 19% 35.1M 5s 2024-04-25T03:24:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33000K .......... .......... .......... .......... .......... 19% 18.6M 5s 2024-04-25T03:24:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33050K .......... .......... .......... .......... .......... 19% 39.3M 5s 2024-04-25T03:24:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33100K .......... .......... .......... .......... .......... 19% 23.7M 5s 2024-04-25T03:24:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33150K .......... .......... .......... .......... .......... 19% 23.9M 5s 2024-04-25T03:24:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33200K .......... .......... .......... .......... .......... 19% 29.2M 5s 2024-04-25T03:24:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33250K .......... .......... .......... .......... .......... 19% 26.2M 5s 2024-04-25T03:24:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33300K .......... .......... .......... .......... .......... 19% 24.3M 5s 2024-04-25T03:24:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33350K .......... .......... .......... .......... .......... 19% 45.2M 5s 2024-04-25T03:24:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33400K .......... .......... .......... .......... .......... 19% 29.9M 5s 2024-04-25T03:24:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33450K .......... .......... .......... .......... .......... 19% 30.6M 5s 2024-04-25T03:24:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33500K .......... .......... .......... .......... .......... 19% 22.5M 5s 2024-04-25T03:24:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33550K .......... .......... .......... .......... .......... 19% 18.3M 5s 2024-04-25T03:24:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33600K .......... .......... .......... .......... .......... 19% 35.8M 5s 2024-04-25T03:24:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33650K .......... .......... .......... .......... .......... 19% 56.4M 5s 2024-04-25T03:24:17.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33700K .......... .......... .......... .......... .......... 20% 23.2M 5s 2024-04-25T03:24:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33750K .......... .......... .......... .......... .......... 20% 32.8M 5s 2024-04-25T03:24:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33800K .......... .......... .......... .......... .......... 20% 25.8M 5s 2024-04-25T03:24:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33850K .......... .......... .......... .......... .......... 20% 41.7M 5s 2024-04-25T03:24:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33900K .......... .......... .......... .......... .......... 20% 27.5M 5s 2024-04-25T03:24:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 33950K .......... .......... .......... .......... .......... 20% 18.6M 5s 2024-04-25T03:24:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34000K .......... .......... .......... .......... .......... 20% 78.8M 5s 2024-04-25T03:24:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34050K .......... .......... .......... .......... .......... 20% 20.0M 5s 2024-04-25T03:24:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34100K .......... .......... .......... .......... .......... 20% 22.6M 5s 2024-04-25T03:24:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34150K .......... .......... .......... .......... .......... 20% 44.4M 5s 2024-04-25T03:24:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34200K .......... .......... .......... .......... .......... 20% 25.5M 5s 2024-04-25T03:24:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34250K .......... .......... .......... .......... .......... 20% 35.9M 5s 2024-04-25T03:24:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34300K .......... .......... .......... .......... .......... 20% 7.53M 5s 2024-04-25T03:24:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34350K .......... .......... .......... .......... .......... 20% 34.7M 5s 2024-04-25T03:24:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34400K .......... .......... .......... .......... .......... 20% 38.8M 5s 2024-04-25T03:24:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34450K .......... .......... .......... .......... .......... 20% 29.2M 5s 2024-04-25T03:24:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34500K .......... .......... .......... .......... .......... 20% 37.9M 5s 2024-04-25T03:24:17.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34550K .......... .......... .......... .......... .......... 20% 21.1M 5s 2024-04-25T03:24:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34600K .......... .......... .......... .......... .......... 20% 44.4M 5s 2024-04-25T03:24:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34650K .......... .......... .......... .......... .......... 20% 40.4M 5s 2024-04-25T03:24:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34700K .......... .......... .......... .......... .......... 20% 23.9M 5s 2024-04-25T03:24:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34750K .......... .......... .......... .......... .......... 20% 22.1M 5s 2024-04-25T03:24:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34800K .......... .......... .......... .......... .......... 20% 27.5M 5s 2024-04-25T03:24:17.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34850K .......... .......... .......... .......... .......... 20% 24.2M 5s 2024-04-25T03:24:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34900K .......... .......... .......... .......... .......... 20% 56.5M 5s 2024-04-25T03:24:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 34950K .......... .......... .......... .......... .......... 20% 24.6M 5s 2024-04-25T03:24:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35000K .......... .......... .......... .......... .......... 20% 25.8M 5s 2024-04-25T03:24:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35050K .......... .......... .......... .......... .......... 20% 34.1M 5s 2024-04-25T03:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35100K .......... .......... .......... .......... .......... 20% 42.7M 5s 2024-04-25T03:24:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35150K .......... .......... .......... .......... .......... 20% 22.9M 5s 2024-04-25T03:24:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35200K .......... .......... .......... .......... .......... 20% 25.6M 5s 2024-04-25T03:24:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35250K .......... .......... .......... .......... .......... 20% 32.3M 5s 2024-04-25T03:24:17.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35300K .......... .......... .......... .......... .......... 20% 38.9M 5s 2024-04-25T03:24:17.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35350K .......... .......... .......... .......... .......... 21% 18.4M 5s 2024-04-25T03:24:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35400K .......... .......... .......... .......... .......... 21% 43.8M 5s 2024-04-25T03:24:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35450K .......... .......... .......... .......... .......... 21% 27.8M 5s 2024-04-25T03:24:17.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35500K .......... .......... .......... .......... .......... 21% 37.8M 5s 2024-04-25T03:24:17.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35550K .......... .......... .......... .......... .......... 21% 23.8M 5s 2024-04-25T03:24:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35600K .......... .......... .......... .......... .......... 21% 23.0M 5s 2024-04-25T03:24:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35650K .......... .......... .......... .......... .......... 21% 61.5M 5s 2024-04-25T03:24:17.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35700K .......... .......... .......... .......... .......... 21% 31.6M 5s 2024-04-25T03:24:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35750K .......... .......... .......... .......... .......... 21% 24.9M 5s 2024-04-25T03:24:17.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35800K .......... .......... .......... .......... .......... 21% 23.4M 5s 2024-04-25T03:24:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35850K .......... .......... .......... .......... .......... 21% 26.6M 5s 2024-04-25T03:24:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35900K .......... .......... .......... .......... .......... 21% 49.7M 5s 2024-04-25T03:24:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 35950K .......... .......... .......... .......... .......... 21% 20.4M 5s 2024-04-25T03:24:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36000K .......... .......... .......... .......... .......... 21% 23.2M 5s 2024-04-25T03:24:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36050K .......... .......... .......... .......... .......... 21% 36.5M 5s 2024-04-25T03:24:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36100K .......... .......... .......... .......... .......... 21% 30.5M 5s 2024-04-25T03:24:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36150K .......... .......... .......... .......... .......... 21% 37.8M 5s 2024-04-25T03:24:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36200K .......... .......... .......... .......... .......... 21% 25.4M 5s 2024-04-25T03:24:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36250K .......... .......... .......... .......... .......... 21% 28.5M 5s 2024-04-25T03:24:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36300K .......... .......... .......... .......... .......... 21% 25.6M 5s 2024-04-25T03:24:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36350K .......... .......... .......... .......... .......... 21% 27.4M 5s 2024-04-25T03:24:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36400K .......... .......... .......... .......... .......... 21% 21.1M 5s 2024-04-25T03:24:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36450K .......... .......... .......... .......... .......... 21% 48.3M 5s 2024-04-25T03:24:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36500K .......... .......... .......... .......... .......... 21% 24.8M 5s 2024-04-25T03:24:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36550K .......... .......... .......... .......... .......... 21% 41.7M 5s 2024-04-25T03:24:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36600K .......... .......... .......... .......... .......... 21% 23.2M 5s 2024-04-25T03:24:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36650K .......... .......... .......... .......... .......... 21% 44.4M 5s 2024-04-25T03:24:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36700K .......... .......... .......... .......... .......... 21% 26.1M 5s 2024-04-25T03:24:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36750K .......... .......... .......... .......... .......... 21% 23.5M 5s 2024-04-25T03:24:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36800K .......... .......... .......... .......... .......... 21% 30.0M 5s 2024-04-25T03:24:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36850K .......... .......... .......... .......... .......... 21% 24.8M 5s 2024-04-25T03:24:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36900K .......... .......... .......... .......... .......... 21% 72.5M 5s 2024-04-25T03:24:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 36950K .......... .......... .......... .......... .......... 21% 24.0M 5s 2024-04-25T03:24:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37000K .......... .......... .......... .......... .......... 21% 36.7M 5s 2024-04-25T03:24:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37050K .......... .......... .......... .......... .......... 22% 23.8M 5s 2024-04-25T03:24:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37100K .......... .......... .......... .......... .......... 22% 30.9M 5s 2024-04-25T03:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37150K .......... .......... .......... .......... .......... 22% 17.8M 5s 2024-04-25T03:24:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37200K .......... .......... .......... .......... .......... 22% 67.8M 5s 2024-04-25T03:24:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37250K .......... .......... .......... .......... .......... 22% 22.9M 5s 2024-04-25T03:24:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37300K .......... .......... .......... .......... .......... 22% 45.2M 5s 2024-04-25T03:24:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37350K .......... .......... .......... .......... .......... 22% 24.4M 5s 2024-04-25T03:24:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37400K .......... .......... .......... .......... .......... 22% 42.4M 5s 2024-04-25T03:24:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37450K .......... .......... .......... .......... .......... 22% 25.3M 5s 2024-04-25T03:24:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37500K .......... .......... .......... .......... .......... 22% 38.0M 5s 2024-04-25T03:24:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37550K .......... .......... .......... .......... .......... 22% 21.5M 5s 2024-04-25T03:24:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37600K .......... .......... .......... .......... .......... 22% 21.3M 5s 2024-04-25T03:24:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37650K .......... .......... .......... .......... .......... 22% 28.0M 5s 2024-04-25T03:24:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37700K .......... .......... .......... .......... .......... 22% 56.8M 5s 2024-04-25T03:24:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37750K .......... .......... .......... .......... .......... 22% 24.2M 5s 2024-04-25T03:24:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37800K .......... .......... .......... .......... .......... 22% 47.1M 5s 2024-04-25T03:24:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37850K .......... .......... .......... .......... .......... 22% 22.3M 5s 2024-04-25T03:24:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37900K .......... .......... .......... .......... .......... 22% 48.8M 5s 2024-04-25T03:24:17.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 37950K .......... .......... .......... .......... .......... 22% 23.6M 5s 2024-04-25T03:24:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38000K .......... .......... .......... .......... .......... 22% 18.2M 5s 2024-04-25T03:24:17.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38050K .......... .......... .......... .......... .......... 22% 39.4M 5s 2024-04-25T03:24:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38100K .......... .......... .......... .......... .......... 22% 35.0M 5s 2024-04-25T03:24:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38150K .......... .......... .......... .......... .......... 22% 21.8M 5s 2024-04-25T03:24:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38200K .......... .......... .......... .......... .......... 22% 59.7M 5s 2024-04-25T03:24:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38250K .......... .......... .......... .......... .......... 22% 24.1M 5s 2024-04-25T03:24:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38300K .......... .......... .......... .......... .......... 22% 32.5M 5s 2024-04-25T03:24:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38350K .......... .......... .......... .......... .......... 22% 26.1M 5s 2024-04-25T03:24:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38400K .......... .......... .......... .......... .......... 22% 21.8M 5s 2024-04-25T03:24:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38450K .......... .......... .......... .......... .......... 22% 42.2M 5s 2024-04-25T03:24:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38500K .......... .......... .......... .......... .......... 22% 38.1M 5s 2024-04-25T03:24:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38550K .......... .......... .......... .......... .......... 22% 21.9M 5s 2024-04-25T03:24:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38600K .......... .......... .......... .......... .......... 22% 26.1M 5s 2024-04-25T03:24:17.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38650K .......... .......... .......... .......... .......... 22% 27.0M 5s 2024-04-25T03:24:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38700K .......... .......... .......... .......... .......... 22% 39.0M 5s 2024-04-25T03:24:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38750K .......... .......... .......... .......... .......... 23% 27.7M 5s 2024-04-25T03:24:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38800K .......... .......... .......... .......... .......... 23% 28.7M 5s 2024-04-25T03:24:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38850K .......... .......... .......... .......... .......... 23% 20.8M 5s 2024-04-25T03:24:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38900K .......... .......... .......... .......... .......... 23% 29.1M 5s 2024-04-25T03:24:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 38950K .......... .......... .......... .......... .......... 23% 71.9M 5s 2024-04-25T03:24:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39000K .......... .......... .......... .......... .......... 23% 30.4M 5s 2024-04-25T03:24:17.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39050K .......... .......... .......... .......... .......... 23% 25.6M 5s 2024-04-25T03:24:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39100K .......... .......... .......... .......... .......... 23% 24.3M 5s 2024-04-25T03:24:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39150K .......... .......... .......... .......... .......... 23% 24.5M 5s 2024-04-25T03:24:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39200K .......... .......... .......... .......... .......... 23% 23.0M 5s 2024-04-25T03:24:17.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39250K .......... .......... .......... .......... .......... 23% 71.5M 5s 2024-04-25T03:24:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39300K .......... .......... .......... .......... .......... 23% 31.5M 5s 2024-04-25T03:24:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39350K .......... .......... .......... .......... .......... 23% 22.1M 5s 2024-04-25T03:24:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39400K .......... .......... .......... .......... .......... 23% 24.4M 5s 2024-04-25T03:24:17.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39450K .......... .......... .......... .......... .......... 23% 65.8M 5s 2024-04-25T03:24:17.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39500K .......... .......... .......... .......... .......... 23% 23.8M 5s 2024-04-25T03:24:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39550K .......... .......... .......... .......... .......... 23% 24.4M 5s 2024-04-25T03:24:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39600K .......... .......... .......... .......... .......... 23% 33.0M 5s 2024-04-25T03:24:17.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39650K .......... .......... .......... .......... .......... 23% 27.4M 5s 2024-04-25T03:24:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39700K .......... .......... .......... .......... .......... 23% 40.2M 5s 2024-04-25T03:24:17.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39750K .......... .......... .......... .......... .......... 23% 28.0M 5s 2024-04-25T03:24:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39800K .......... .......... .......... .......... .......... 23% 30.1M 5s 2024-04-25T03:24:17.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39850K .......... .......... .......... .......... .......... 23% 23.4M 5s 2024-04-25T03:24:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39900K .......... .......... .......... .......... .......... 23% 21.3M 5s 2024-04-25T03:24:17.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 39950K .......... .......... .......... .......... .......... 23% 27.2M 5s 2024-04-25T03:24:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40000K .......... .......... .......... .......... .......... 23% 53.1M 5s 2024-04-25T03:24:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40050K .......... .......... .......... .......... .......... 23% 24.2M 5s 2024-04-25T03:24:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40100K .......... .......... .......... .......... .......... 23% 31.5M 5s 2024-04-25T03:24:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40150K .......... .......... .......... .......... .......... 23% 17.9M 5s 2024-04-25T03:24:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40200K .......... .......... .......... .......... .......... 23% 188M 5s 2024-04-25T03:24:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40250K .......... .......... .......... .......... .......... 23% 24.7M 5s 2024-04-25T03:24:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40300K .......... .......... .......... .......... .......... 23% 44.4M 5s 2024-04-25T03:24:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40350K .......... .......... .......... .......... .......... 23% 22.1M 5s 2024-04-25T03:24:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40400K .......... .......... .......... .......... .......... 23% 22.7M 5s 2024-04-25T03:24:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40450K .......... .......... .......... .......... .......... 24% 21.5M 5s 2024-04-25T03:24:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40500K .......... .......... .......... .......... .......... 24% 59.3M 5s 2024-04-25T03:24:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40550K .......... .......... .......... .......... .......... 24% 19.7M 5s 2024-04-25T03:24:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40600K .......... .......... .......... .......... .......... 24% 30.1M 5s 2024-04-25T03:24:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40650K .......... .......... .......... .......... .......... 24% 34.8M 5s 2024-04-25T03:24:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40700K .......... .......... .......... .......... .......... 24% 61.9M 5s 2024-04-25T03:24:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40750K .......... .......... .......... .......... .......... 24% 21.4M 5s 2024-04-25T03:24:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40800K .......... .......... .......... .......... .......... 24% 25.9M 5s 2024-04-25T03:24:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40850K .......... .......... .......... .......... .......... 24% 34.9M 5s 2024-04-25T03:24:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40900K .......... .......... .......... .......... .......... 24% 24.0M 5s 2024-04-25T03:24:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 40950K .......... .......... .......... .......... .......... 24% 22.0M 5s 2024-04-25T03:24:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41000K .......... .......... .......... .......... .......... 24% 68.8M 5s 2024-04-25T03:24:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41050K .......... .......... .......... .......... .......... 24% 35.4M 5s 2024-04-25T03:24:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41100K .......... .......... .......... .......... .......... 24% 19.8M 5s 2024-04-25T03:24:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41150K .......... .......... .......... .......... .......... 24% 29.9M 5s 2024-04-25T03:24:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41200K .......... .......... .......... .......... .......... 24% 26.5M 5s 2024-04-25T03:24:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41250K .......... .......... .......... .......... .......... 24% 49.6M 5s 2024-04-25T03:24:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41300K .......... .......... .......... .......... .......... 24% 23.4M 5s 2024-04-25T03:24:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41350K .......... .......... .......... .......... .......... 24% 24.4M 5s 2024-04-25T03:24:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41400K .......... .......... .......... .......... .......... 24% 27.4M 5s 2024-04-25T03:24:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41450K .......... .......... .......... .......... .......... 24% 45.7M 5s 2024-04-25T03:24:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41500K .......... .......... .......... .......... .......... 24% 29.6M 5s 2024-04-25T03:24:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41550K .......... .......... .......... .......... .......... 24% 23.0M 5s 2024-04-25T03:24:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41600K .......... .......... .......... .......... .......... 24% 30.5M 5s 2024-04-25T03:24:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41650K .......... .......... .......... .......... .......... 24% 18.0M 5s 2024-04-25T03:24:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41700K .......... .......... .......... .......... .......... 24% 26.3M 5s 2024-04-25T03:24:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41750K .......... .......... .......... .......... .......... 24% 40.3M 5s 2024-04-25T03:24:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41800K .......... .......... .......... .......... .......... 24% 24.3M 5s 2024-04-25T03:24:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41850K .......... .......... .......... .......... .......... 24% 38.7M 5s 2024-04-25T03:24:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41900K .......... .......... .......... .......... .......... 24% 24.6M 5s 2024-04-25T03:24:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 41950K .......... .......... .......... .......... .......... 24% 24.8M 5s 2024-04-25T03:24:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42000K .......... .......... .......... .......... .......... 24% 62.0M 5s 2024-04-25T03:24:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42050K .......... .......... .......... .......... .......... 24% 21.3M 5s 2024-04-25T03:24:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42100K .......... .......... .......... .......... .......... 25% 38.8M 5s 2024-04-25T03:24:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42150K .......... .......... .......... .......... .......... 25% 20.5M 5s 2024-04-25T03:24:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42200K .......... .......... .......... .......... .......... 25% 27.6M 5s 2024-04-25T03:24:18.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42250K .......... .......... .......... .......... .......... 25% 51.5M 5s 2024-04-25T03:24:18.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42300K .......... .......... .......... .......... .......... 25% 32.1M 5s 2024-04-25T03:24:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42350K .......... .......... .......... .......... .......... 25% 20.3M 5s 2024-04-25T03:24:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42400K .......... .......... .......... .......... .......... 25% 31.0M 5s 2024-04-25T03:24:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42450K .......... .......... .......... .......... .......... 25% 26.1M 5s 2024-04-25T03:24:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42500K .......... .......... .......... .......... .......... 25% 53.8M 5s 2024-04-25T03:24:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42550K .......... .......... .......... .......... .......... 25% 24.2M 5s 2024-04-25T03:24:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42600K .......... .......... .......... .......... .......... 25% 29.8M 5s 2024-04-25T03:24:18.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42650K .......... .......... .......... .......... .......... 25% 24.0M 5s 2024-04-25T03:24:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42700K .......... .......... .......... .......... .......... 25% 29.0M 5s 2024-04-25T03:24:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42750K .......... .......... .......... .......... .......... 25% 23.9M 5s 2024-04-25T03:24:18.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42800K .......... .......... .......... .......... .......... 25% 47.6M 5s 2024-04-25T03:24:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42850K .......... .......... .......... .......... .......... 25% 22.8M 5s 2024-04-25T03:24:18.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42900K .......... .......... .......... .......... .......... 25% 33.2M 5s 2024-04-25T03:24:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 42950K .......... .......... .......... .......... .......... 25% 22.8M 5s 2024-04-25T03:24:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43000K .......... .......... .......... .......... .......... 25% 76.1M 5s 2024-04-25T03:24:18.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43050K .......... .......... .......... .......... .......... 25% 24.9M 5s 2024-04-25T03:24:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43100K .......... .......... .......... .......... .......... 25% 25.0M 5s 2024-04-25T03:24:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43150K .......... .......... .......... .......... .......... 25% 33.7M 5s 2024-04-25T03:24:18.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43200K .......... .......... .......... .......... .......... 25% 23.6M 5s 2024-04-25T03:24:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43250K .......... .......... .......... .......... .......... 25% 23.4M 5s 2024-04-25T03:24:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43300K .......... .......... .......... .......... .......... 25% 55.0M 5s 2024-04-25T03:24:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43350K .......... .......... .......... .......... .......... 25% 20.5M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43400K .......... .......... .......... .......... .......... 25% 588K 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43450K .......... .......... .......... .......... .......... 25% 152M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43500K .......... .......... .......... .......... .......... 25% 191M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43550K .......... .......... .......... .......... .......... 25% 166M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43600K .......... .......... .......... .......... .......... 25% 173M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43650K .......... .......... .......... .......... .......... 25% 162M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43700K .......... .......... .......... .......... .......... 25% 198M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43750K .......... .......... .......... .......... .......... 25% 8.76M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43800K .......... .......... .......... .......... .......... 26% 152M 5s 2024-04-25T03:24:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43850K .......... .......... .......... .......... .......... 26% 189M 5s 2024-04-25T03:24:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43900K .......... .......... .......... .......... .......... 26% 175M 5s 2024-04-25T03:24:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 43950K .......... .......... .......... .......... .......... 26% 144M 5s 2024-04-25T03:24:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44000K .......... .......... .......... .......... .......... 26% 170M 5s 2024-04-25T03:24:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44050K .......... .......... .......... .......... .......... 26% 174M 5s 2024-04-25T03:24:18.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44100K .......... .......... .......... .......... .......... 26% 165M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44150K .......... .......... .......... .......... .......... 26% 6.60M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44200K .......... .......... .......... .......... .......... 26% 191M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44250K .......... .......... .......... .......... .......... 26% 172M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44300K .......... .......... .......... .......... .......... 26% 187M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44350K .......... .......... .......... .......... .......... 26% 164M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44400K .......... .......... .......... .......... .......... 26% 189M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44450K .......... .......... .......... .......... .......... 26% 156M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44500K .......... .......... .......... .......... .......... 26% 169M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44550K .......... .......... .......... .......... .......... 26% 175M 5s 2024-04-25T03:24:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44600K .......... .......... .......... .......... .......... 26% 164M 5s 2024-04-25T03:24:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44650K .......... .......... .......... .......... .......... 26% 166M 5s 2024-04-25T03:24:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44700K .......... .......... .......... .......... .......... 26% 3.67M 5s 2024-04-25T03:24:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44750K .......... .......... .......... .......... .......... 26% 135M 5s 2024-04-25T03:24:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44800K .......... .......... .......... .......... .......... 26% 172M 5s 2024-04-25T03:24:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44850K .......... .......... .......... .......... .......... 26% 161M 5s 2024-04-25T03:24:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44900K .......... .......... .......... .......... .......... 26% 10.6M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 44950K .......... .......... .......... .......... .......... 26% 171M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45000K .......... .......... .......... .......... .......... 26% 191M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45050K .......... .......... .......... .......... .......... 26% 180M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45100K .......... .......... .......... .......... .......... 26% 173M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45150K .......... .......... .......... .......... .......... 26% 132M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45200K .......... .......... .......... .......... .......... 26% 197M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45250K .......... .......... .......... .......... .......... 26% 170M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45300K .......... .......... .......... .......... .......... 26% 175M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45350K .......... .......... .......... .......... .......... 26% 157M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45400K .......... .......... .......... .......... .......... 26% 181M 5s 2024-04-25T03:24:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45450K .......... .......... .......... .......... .......... 26% 175M 5s 2024-04-25T03:24:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45500K .......... .......... .......... .......... .......... 27% 4.69M 5s 2024-04-25T03:24:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45550K .......... .......... .......... .......... .......... 27% 139M 5s 2024-04-25T03:24:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45600K .......... .......... .......... .......... .......... 27% 188M 5s 2024-04-25T03:24:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45650K .......... .......... .......... .......... .......... 27% 12.3M 5s 2024-04-25T03:24:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45700K .......... .......... .......... .......... .......... 27% 172M 5s 2024-04-25T03:24:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45750K .......... .......... .......... .......... .......... 27% 168M 5s 2024-04-25T03:24:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45800K .......... .......... .......... .......... .......... 27% 151M 5s 2024-04-25T03:24:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45850K .......... .......... .......... .......... .......... 27% 184M 5s 2024-04-25T03:24:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45900K .......... .......... .......... .......... .......... 27% 199M 5s 2024-04-25T03:24:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 45950K .......... .......... .......... .......... .......... 27% 153M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46000K .......... .......... .......... .......... .......... 27% 189M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46050K .......... .......... .......... .......... .......... 27% 155M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46100K .......... .......... .......... .......... .......... 27% 184M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46150K .......... .......... .......... .......... .......... 27% 146M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46200K .......... .......... .......... .......... .......... 27% 154M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46250K .......... .......... .......... .......... .......... 27% 163M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46300K .......... .......... .......... .......... .......... 27% 164M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46350K .......... .......... .......... .......... .......... 27% 159M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46400K .......... .......... .......... .......... .......... 27% 141M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46450K .......... .......... .......... .......... .......... 27% 185M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46500K .......... .......... .......... .......... .......... 27% 187M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46550K .......... .......... .......... .......... .......... 27% 138M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46600K .......... .......... .......... .......... .......... 27% 180M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46650K .......... .......... .......... .......... .......... 27% 197M 5s 2024-04-25T03:24:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46700K .......... .......... .......... .......... .......... 27% 197M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46750K .......... .......... .......... .......... .......... 27% 140M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46800K .......... .......... .......... .......... .......... 27% 184M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46850K .......... .......... .......... .......... .......... 27% 189M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46900K .......... .......... .......... .......... .......... 27% 202M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 46950K .......... .......... .......... .......... .......... 27% 136M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47000K .......... .......... .......... .......... .......... 27% 166M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47050K .......... .......... .......... .......... .......... 27% 191M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47100K .......... .......... .......... .......... .......... 27% 173M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47150K .......... .......... .......... .......... .......... 28% 146M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47200K .......... .......... .......... .......... .......... 28% 180M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47250K .......... .......... .......... .......... .......... 28% 196M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47300K .......... .......... .......... .......... .......... 28% 153M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47350K .......... .......... .......... .......... .......... 28% 143M 5s 2024-04-25T03:24:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47400K .......... .......... .......... .......... .......... 28% 201M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47450K .......... .......... .......... .......... .......... 28% 203M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47500K .......... .......... .......... .......... .......... 28% 144M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47550K .......... .......... .......... .......... .......... 28% 13.8M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47600K .......... .......... .......... .......... .......... 28% 125M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47650K .......... .......... .......... .......... .......... 28% 29.4M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47700K .......... .......... .......... .......... .......... 28% 23.5M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47750K .......... .......... .......... .......... .......... 28% 65.0M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47800K .......... .......... .......... .......... .......... 28% 18.2M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47850K .......... .......... .......... .......... .......... 28% 63.1M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47900K .......... .......... .......... .......... .......... 28% 23.8M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 47950K .......... .......... .......... .......... .......... 28% 23.6M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48000K .......... .......... .......... .......... .......... 28% 37.0M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48050K .......... .......... .......... .......... .......... 28% 24.3M 5s 2024-04-25T03:24:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48100K .......... .......... .......... .......... .......... 28% 50.4M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48150K .......... .......... .......... .......... .......... 28% 22.2M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48200K .......... .......... .......... .......... .......... 28% 29.5M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48250K .......... .......... .......... .......... .......... 28% 35.6M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48300K .......... .......... .......... .......... .......... 28% 24.4M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48350K .......... .......... .......... .......... .......... 28% 24.1M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48400K .......... .......... .......... .......... .......... 28% 54.2M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48450K .......... .......... .......... .......... .......... 28% 24.6M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48500K .......... .......... .......... .......... .......... 28% 32.4M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48550K .......... .......... .......... .......... .......... 28% 23.9M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48600K .......... .......... .......... .......... .......... 28% 53.3M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48650K .......... .......... .......... .......... .......... 28% 26.2M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48700K .......... .......... .......... .......... .......... 28% 21.5M 5s 2024-04-25T03:24:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48750K .......... .......... .......... .......... .......... 28% 33.3M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48800K .......... .......... .......... .......... .......... 28% 25.4M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48850K .......... .......... .......... .......... .......... 29% 19.0M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48900K .......... .......... .......... .......... .......... 29% 148M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 48950K .......... .......... .......... .......... .......... 29% 21.4M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49000K .......... .......... .......... .......... .......... 29% 30.8M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49050K .......... .......... .......... .......... .......... 29% 19.6M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49100K .......... .......... .......... .......... .......... 29% 161M 5s 2024-04-25T03:24:18.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49150K .......... .......... .......... .......... .......... 29% 26.6M 5s 2024-04-25T03:24:18.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49200K .......... .......... .......... .......... .......... 29% 20.8M 5s 2024-04-25T03:24:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49250K .......... .......... .......... .......... .......... 29% 22.5M 5s 2024-04-25T03:24:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49300K .......... .......... .......... .......... .......... 29% 72.9M 5s 2024-04-25T03:24:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49350K .......... .......... .......... .......... .......... 29% 41.8M 5s 2024-04-25T03:24:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49400K .......... .......... .......... .......... .......... 29% 24.8M 5s 2024-04-25T03:24:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49450K .......... .......... .......... .......... .......... 29% 27.1M 5s 2024-04-25T03:24:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49500K .......... .......... .......... .......... .......... 29% 19.6M 5s 2024-04-25T03:24:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49550K .......... .......... .......... .......... .......... 29% 28.7M 5s 2024-04-25T03:24:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49600K .......... .......... .......... .......... .......... 29% 24.5M 5s 2024-04-25T03:24:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49650K .......... .......... .......... .......... .......... 29% 149M 5s 2024-04-25T03:24:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49700K .......... .......... .......... .......... .......... 29% 21.9M 5s 2024-04-25T03:24:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49750K .......... .......... .......... .......... .......... 29% 29.8M 4s 2024-04-25T03:24:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49800K .......... .......... .......... .......... .......... 29% 24.0M 4s 2024-04-25T03:24:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49850K .......... .......... .......... .......... .......... 29% 112M 4s 2024-04-25T03:24:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49900K .......... .......... .......... .......... .......... 29% 26.1M 4s 2024-04-25T03:24:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 49950K .......... .......... .......... .......... .......... 29% 21.0M 4s 2024-04-25T03:24:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50000K .......... .......... .......... .......... .......... 29% 26.2M 4s 2024-04-25T03:24:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50050K .......... .......... .......... .......... .......... 29% 26.5M 4s 2024-04-25T03:24:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50100K .......... .......... .......... .......... .......... 29% 22.0M 4s 2024-04-25T03:24:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50150K .......... .......... .......... .......... .......... 29% 47.2M 4s 2024-04-25T03:24:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50200K .......... .......... .......... .......... .......... 29% 28.7M 4s 2024-04-25T03:24:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50250K .......... .......... .......... .......... .......... 29% 32.7M 4s 2024-04-25T03:24:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50300K .......... .......... .......... .......... .......... 29% 22.9M 4s 2024-04-25T03:24:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50350K .......... .......... .......... .......... .......... 29% 24.9M 4s 2024-04-25T03:24:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50400K .......... .......... .......... .......... .......... 29% 51.8M 4s 2024-04-25T03:24:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50450K .......... .......... .......... .......... .......... 29% 30.6M 4s 2024-04-25T03:24:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50500K .......... .......... .......... .......... .......... 29% 24.7M 4s 2024-04-25T03:24:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50550K .......... .......... .......... .......... .......... 30% 21.0M 4s 2024-04-25T03:24:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50600K .......... .......... .......... .......... .......... 30% 30.2M 4s 2024-04-25T03:24:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50650K .......... .......... .......... .......... .......... 30% 66.9M 4s 2024-04-25T03:24:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50700K .......... .......... .......... .......... .......... 30% 19.5M 4s 2024-04-25T03:24:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50750K .......... .......... .......... .......... .......... 30% 24.7M 4s 2024-04-25T03:24:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50800K .......... .......... .......... .......... .......... 30% 41.5M 4s 2024-04-25T03:24:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50850K .......... .......... .......... .......... .......... 30% 20.0M 4s 2024-04-25T03:24:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50900K .......... .......... .......... .......... .......... 30% 54.7M 4s 2024-04-25T03:24:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 50950K .......... .......... .......... .......... .......... 30% 23.8M 4s 2024-04-25T03:24:18.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51000K .......... .......... .......... .......... .......... 30% 36.0M 4s 2024-04-25T03:24:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51050K .......... .......... .......... .......... .......... 30% 25.7M 4s 2024-04-25T03:24:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51100K .......... .......... .......... .......... .......... 30% 16.4M 4s 2024-04-25T03:24:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51150K .......... .......... .......... .......... .......... 30% 21.4M 4s 2024-04-25T03:24:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51200K .......... .......... .......... .......... .......... 30% 69.0M 4s 2024-04-25T03:24:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51250K .......... .......... .......... .......... .......... 30% 22.0M 4s 2024-04-25T03:24:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51300K .......... .......... .......... .......... .......... 30% 40.7M 4s 2024-04-25T03:24:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51350K .......... .......... .......... .......... .......... 30% 22.3M 4s 2024-04-25T03:24:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51400K .......... .......... .......... .......... .......... 30% 45.5M 4s 2024-04-25T03:24:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51450K .......... .......... .......... .......... .......... 30% 11.6M 4s 2024-04-25T03:24:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51500K .......... .......... .......... .......... .......... 30% 33.5M 4s 2024-04-25T03:24:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51550K .......... .......... .......... .......... .......... 30% 144M 4s 2024-04-25T03:24:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51600K .......... .......... .......... .......... .......... 30% 47.9M 4s 2024-04-25T03:24:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51650K .......... .......... .......... .......... .......... 30% 21.5M 4s 2024-04-25T03:24:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51700K .......... .......... .......... .......... .......... 30% 46.3M 4s 2024-04-25T03:24:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51750K .......... .......... .......... .......... .......... 30% 21.8M 4s 2024-04-25T03:24:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51800K .......... .......... .......... .......... .......... 30% 40.9M 4s 2024-04-25T03:24:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51850K .......... .......... .......... .......... .......... 30% 24.4M 4s 2024-04-25T03:24:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51900K .......... .......... .......... .......... .......... 30% 42.6M 4s 2024-04-25T03:24:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 51950K .......... .......... .......... .......... .......... 30% 27.2M 4s 2024-04-25T03:24:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52000K .......... .......... .......... .......... .......... 30% 24.8M 4s 2024-04-25T03:24:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52050K .......... .......... .......... .......... .......... 30% 40.0M 4s 2024-04-25T03:24:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52100K .......... .......... .......... .......... .......... 30% 23.5M 4s 2024-04-25T03:24:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52150K .......... .......... .......... .......... .......... 30% 47.0M 4s 2024-04-25T03:24:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52200K .......... .......... .......... .......... .......... 30% 24.3M 4s 2024-04-25T03:24:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52250K .......... .......... .......... .......... .......... 31% 35.5M 4s 2024-04-25T03:24:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52300K .......... .......... .......... .......... .......... 31% 21.5M 4s 2024-04-25T03:24:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52350K .......... .......... .......... .......... .......... 31% 27.8M 4s 2024-04-25T03:24:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52400K .......... .......... .......... .......... .......... 31% 21.4M 4s 2024-04-25T03:24:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52450K .......... .......... .......... .......... .......... 31% 49.0M 4s 2024-04-25T03:24:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52500K .......... .......... .......... .......... .......... 31% 24.4M 4s 2024-04-25T03:24:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52550K .......... .......... .......... .......... .......... 31% 37.7M 4s 2024-04-25T03:24:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52600K .......... .......... .......... .......... .......... 31% 21.2M 4s 2024-04-25T03:24:18.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52650K .......... .......... .......... .......... .......... 31% 52.6M 4s 2024-04-25T03:24:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52700K .......... .......... .......... .......... .......... 31% 24.2M 4s 2024-04-25T03:24:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52750K .......... .......... .......... .......... .......... 31% 23.1M 4s 2024-04-25T03:24:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52800K .......... .......... .......... .......... .......... 31% 41.8M 4s 2024-04-25T03:24:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52850K .......... .......... .......... .......... .......... 31% 23.9M 4s 2024-04-25T03:24:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52900K .......... .......... .......... .......... .......... 31% 22.7M 4s 2024-04-25T03:24:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 52950K .......... .......... .......... .......... .......... 31% 50.3M 4s 2024-04-25T03:24:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53000K .......... .......... .......... .......... .......... 31% 26.8M 4s 2024-04-25T03:24:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53050K .......... .......... .......... .......... .......... 31% 33.7M 4s 2024-04-25T03:24:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53100K .......... .......... .......... .......... .......... 31% 24.8M 4s 2024-04-25T03:24:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53150K .......... .......... .......... .......... .......... 31% 36.9M 4s 2024-04-25T03:24:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53200K .......... .......... .......... .......... .......... 31% 25.0M 4s 2024-04-25T03:24:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53250K .......... .......... .......... .......... .......... 31% 28.5M 4s 2024-04-25T03:24:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53300K .......... .......... .......... .......... .......... 31% 24.6M 4s 2024-04-25T03:24:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53350K .......... .......... .......... .......... .......... 31% 28.7M 4s 2024-04-25T03:24:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53400K .......... .......... .......... .......... .......... 31% 22.2M 4s 2024-04-25T03:24:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53450K .......... .......... .......... .......... .......... 31% 16.4M 4s 2024-04-25T03:24:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53500K .......... .......... .......... .......... .......... 31% 27.6M 4s 2024-04-25T03:24:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53550K .......... .......... .......... .......... .......... 31% 21.8M 4s 2024-04-25T03:24:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53600K .......... .......... .......... .......... .......... 31% 42.5M 4s 2024-04-25T03:24:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53650K .......... .......... .......... .......... .......... 31% 19.7M 4s 2024-04-25T03:24:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53700K .......... .......... .......... .......... .......... 31% 37.7M 4s 2024-04-25T03:24:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53750K .......... .......... .......... .......... .......... 31% 25.1M 4s 2024-04-25T03:24:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53800K .......... .......... .......... .......... .......... 31% 46.9M 4s 2024-04-25T03:24:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53850K .......... .......... .......... .......... .......... 31% 23.9M 4s 2024-04-25T03:24:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53900K .......... .......... .......... .......... .......... 32% 49.4M 4s 2024-04-25T03:24:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 53950K .......... .......... .......... .......... .......... 32% 15.8M 4s 2024-04-25T03:24:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54000K .......... .......... .......... .......... .......... 32% 92.2M 4s 2024-04-25T03:24:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54050K .......... .......... .......... .......... .......... 32% 17.5M 4s 2024-04-25T03:24:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54100K .......... .......... .......... .......... .......... 32% 32.7M 4s 2024-04-25T03:24:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54150K .......... .......... .......... .......... .......... 32% 23.0M 4s 2024-04-25T03:24:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54200K .......... .......... .......... .......... .......... 32% 27.7M 4s 2024-04-25T03:24:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54250K .......... .......... .......... .......... .......... 32% 25.5M 4s 2024-04-25T03:24:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54300K .......... .......... .......... .......... .......... 32% 36.3M 4s 2024-04-25T03:24:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54350K .......... .......... .......... .......... .......... 32% 23.9M 4s 2024-04-25T03:24:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54400K .......... .......... .......... .......... .......... 32% 18.6M 4s 2024-04-25T03:24:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54450K .......... .......... .......... .......... .......... 32% 48.8M 4s 2024-04-25T03:24:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54500K .......... .......... .......... .......... .......... 32% 23.4M 4s 2024-04-25T03:24:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54550K .......... .......... .......... .......... .......... 32% 24.7M 4s 2024-04-25T03:24:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54600K .......... .......... .......... .......... .......... 32% 29.0M 4s 2024-04-25T03:24:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54650K .......... .......... .......... .......... .......... 32% 24.0M 4s 2024-04-25T03:24:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54700K .......... .......... .......... .......... .......... 32% 67.3M 4s 2024-04-25T03:24:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54750K .......... .......... .......... .......... .......... 32% 23.0M 4s 2024-04-25T03:24:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54800K .......... .......... .......... .......... .......... 32% 23.5M 4s 2024-04-25T03:24:18.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54850K .......... .......... .......... .......... .......... 32% 36.6M 4s 2024-04-25T03:24:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54900K .......... .......... .......... .......... .......... 32% 26.2M 4s 2024-04-25T03:24:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 54950K .......... .......... .......... .......... .......... 32% 35.4M 4s 2024-04-25T03:24:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55000K .......... .......... .......... .......... .......... 32% 27.4M 4s 2024-04-25T03:24:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55050K .......... .......... .......... .......... .......... 32% 40.6M 4s 2024-04-25T03:24:18.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55100K .......... .......... .......... .......... .......... 32% 19.7M 4s 2024-04-25T03:24:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55150K .......... .......... .......... .......... .......... 32% 27.1M 4s 2024-04-25T03:24:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55200K .......... .......... .......... .......... .......... 32% 23.5M 4s 2024-04-25T03:24:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55250K .......... .......... .......... .......... .......... 32% 37.8M 4s 2024-04-25T03:24:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55300K .......... .......... .......... .......... .......... 32% 25.5M 4s 2024-04-25T03:24:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55350K .......... .......... .......... .......... .......... 32% 35.1M 4s 2024-04-25T03:24:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55400K .......... .......... .......... .......... .......... 32% 24.4M 4s 2024-04-25T03:24:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55450K .......... .......... .......... .......... .......... 32% 50.0M 4s 2024-04-25T03:24:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55500K .......... .......... .......... .......... .......... 32% 20.4M 4s 2024-04-25T03:24:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55550K .......... .......... .......... .......... .......... 32% 20.6M 4s 2024-04-25T03:24:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55600K .......... .......... .......... .......... .......... 33% 40.7M 4s 2024-04-25T03:24:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55650K .......... .......... .......... .......... .......... 33% 30.7M 4s 2024-04-25T03:24:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55700K .......... .......... .......... .......... .......... 33% 19.8M 4s 2024-04-25T03:24:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55750K .......... .......... .......... .......... .......... 33% 42.9M 4s 2024-04-25T03:24:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55800K .......... .......... .......... .......... .......... 33% 24.4M 4s 2024-04-25T03:24:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55850K .......... .......... .......... .......... .......... 33% 32.2M 4s 2024-04-25T03:24:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55900K .......... .......... .......... .......... .......... 33% 27.4M 4s 2024-04-25T03:24:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 55950K .......... .......... .......... .......... .......... 33% 24.4M 4s 2024-04-25T03:24:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56000K .......... .......... .......... .......... .......... 33% 28.0M 4s 2024-04-25T03:24:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56050K .......... .......... .......... .......... .......... 33% 28.4M 4s 2024-04-25T03:24:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56100K .......... .......... .......... .......... .......... 33% 39.6M 4s 2024-04-25T03:24:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56150K .......... .......... .......... .......... .......... 33% 19.2M 4s 2024-04-25T03:24:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56200K .......... .......... .......... .......... .......... 33% 20.6M 4s 2024-04-25T03:24:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56250K .......... .......... .......... .......... .......... 33% 47.9M 4s 2024-04-25T03:24:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56300K .......... .......... .......... .......... .......... 33% 23.8M 4s 2024-04-25T03:24:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56350K .......... .......... .......... .......... .......... 33% 12.9M 4s 2024-04-25T03:24:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56400K .......... .......... .......... .......... .......... 33% 49.1M 4s 2024-04-25T03:24:18.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56450K .......... .......... .......... .......... .......... 33% 36.6M 4s 2024-04-25T03:24:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56500K .......... .......... .......... .......... .......... 33% 25.4M 4s 2024-04-25T03:24:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56550K .......... .......... .......... .......... .......... 33% 22.6M 4s 2024-04-25T03:24:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56600K .......... .......... .......... .......... .......... 33% 43.3M 4s 2024-04-25T03:24:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56650K .......... .......... .......... .......... .......... 33% 23.2M 4s 2024-04-25T03:24:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56700K .......... .......... .......... .......... .......... 33% 44.1M 4s 2024-04-25T03:24:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56750K .......... .......... .......... .......... .......... 33% 16.2M 4s 2024-04-25T03:24:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56800K .......... .......... .......... .......... .......... 33% 48.5M 4s 2024-04-25T03:24:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56850K .......... .......... .......... .......... .......... 33% 22.3M 4s 2024-04-25T03:24:18.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56900K .......... .......... .......... .......... .......... 33% 39.0M 4s 2024-04-25T03:24:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 56950K .......... .......... .......... .......... .......... 33% 26.3M 4s 2024-04-25T03:24:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57000K .......... .......... .......... .......... .......... 33% 49.4M 4s 2024-04-25T03:24:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57050K .......... .......... .......... .......... .......... 33% 24.4M 4s 2024-04-25T03:24:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57100K .......... .......... .......... .......... .......... 33% 23.7M 4s 2024-04-25T03:24:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57150K .......... .......... .......... .......... .......... 33% 33.3M 4s 2024-04-25T03:24:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57200K .......... .......... .......... .......... .......... 33% 26.3M 4s 2024-04-25T03:24:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57250K .......... .......... .......... .......... .......... 33% 49.2M 4s 2024-04-25T03:24:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57300K .......... .......... .......... .......... .......... 34% 24.4M 4s 2024-04-25T03:24:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57350K .......... .......... .......... .......... .......... 34% 23.3M 4s 2024-04-25T03:24:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57400K .......... .......... .......... .......... .......... 34% 40.4M 4s 2024-04-25T03:24:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57450K .......... .......... .......... .......... .......... 34% 36.2M 4s 2024-04-25T03:24:18.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57500K .......... .......... .......... .......... .......... 34% 29.5M 4s 2024-04-25T03:24:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57550K .......... .......... .......... .......... .......... 34% 22.3M 4s 2024-04-25T03:24:18.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57600K .......... .......... .......... .......... .......... 34% 23.7M 4s 2024-04-25T03:24:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57650K .......... .......... .......... .......... .......... 34% 40.6M 4s 2024-04-25T03:24:18.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57700K .......... .......... .......... .......... .......... 34% 24.0M 4s 2024-04-25T03:24:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57750K .......... .......... .......... .......... .......... 34% 42.3M 4s 2024-04-25T03:24:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57800K .......... .......... .......... .......... .......... 34% 21.6M 4s 2024-04-25T03:24:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57850K .......... .......... .......... .......... .......... 34% 27.9M 4s 2024-04-25T03:24:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57900K .......... .......... .......... .......... .......... 34% 32.5M 4s 2024-04-25T03:24:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 57950K .......... .......... .......... .......... .......... 34% 7.95M 4s 2024-04-25T03:24:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58000K .......... .......... .......... .......... .......... 34% 24.4M 4s 2024-04-25T03:24:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58050K .......... .......... .......... .......... .......... 34% 41.1M 4s 2024-04-25T03:24:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58100K .......... .......... .......... .......... .......... 34% 25.2M 4s 2024-04-25T03:24:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58150K .......... .......... .......... .......... .......... 34% 37.4M 4s 2024-04-25T03:24:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58200K .......... .......... .......... .......... .......... 34% 25.2M 4s 2024-04-25T03:24:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58250K .......... .......... .......... .......... .......... 34% 45.6M 4s 2024-04-25T03:24:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58300K .......... .......... .......... .......... .......... 34% 24.6M 4s 2024-04-25T03:24:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58350K .......... .......... .......... .......... .......... 34% 21.4M 4s 2024-04-25T03:24:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58400K .......... .......... .......... .......... .......... 34% 38.2M 4s 2024-04-25T03:24:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58450K .......... .......... .......... .......... .......... 34% 23.6M 4s 2024-04-25T03:24:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58500K .......... .......... .......... .......... .......... 34% 25.6M 4s 2024-04-25T03:24:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58550K .......... .......... .......... .......... .......... 34% 16.7M 4s 2024-04-25T03:24:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58600K .......... .......... .......... .......... .......... 34% 23.6M 4s 2024-04-25T03:24:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58650K .......... .......... .......... .......... .......... 34% 42.4M 4s 2024-04-25T03:24:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58700K .......... .......... .......... .......... .......... 34% 24.1M 4s 2024-04-25T03:24:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58750K .......... .......... .......... .......... .......... 34% 22.8M 4s 2024-04-25T03:24:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58800K .......... .......... .......... .......... .......... 34% 46.9M 4s 2024-04-25T03:24:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58850K .......... .......... .......... .......... .......... 34% 23.9M 4s 2024-04-25T03:24:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58900K .......... .......... .......... .......... .......... 34% 41.4M 4s 2024-04-25T03:24:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 58950K .......... .......... .......... .......... .......... 35% 24.4M 4s 2024-04-25T03:24:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59000K .......... .......... .......... .......... .......... 35% 43.9M 4s 2024-04-25T03:24:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59050K .......... .......... .......... .......... .......... 35% 26.6M 4s 2024-04-25T03:24:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59100K .......... .......... .......... .......... .......... 35% 19.6M 4s 2024-04-25T03:24:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59150K .......... .......... .......... .......... .......... 35% 25.7M 4s 2024-04-25T03:24:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59200K .......... .......... .......... .......... .......... 35% 35.1M 4s 2024-04-25T03:24:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59250K .......... .......... .......... .......... .......... 35% 27.1M 4s 2024-04-25T03:24:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59300K .......... .......... .......... .......... .......... 35% 35.0M 4s 2024-04-25T03:24:18.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59350K .......... .......... .......... .......... .......... 35% 37.1M 4s 2024-04-25T03:24:18.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59400K .......... .......... .......... .......... .......... 35% 31.9M 4s 2024-04-25T03:24:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59450K .......... .......... .......... .......... .......... 35% 21.7M 4s 2024-04-25T03:24:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59500K .......... .......... .......... .......... .......... 35% 43.7M 4s 2024-04-25T03:24:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59550K .......... .......... .......... .......... .......... 35% 27.7M 4s 2024-04-25T03:24:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59600K .......... .......... .......... .......... .......... 35% 23.1M 4s 2024-04-25T03:24:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59650K .......... .......... .......... .......... .......... 35% 23.0M 4s 2024-04-25T03:24:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59700K .......... .......... .......... .......... .......... 35% 39.9M 4s 2024-04-25T03:24:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59750K .......... .......... .......... .......... .......... 35% 23.6M 4s 2024-04-25T03:24:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59800K .......... .......... .......... .......... .......... 35% 41.8M 4s 2024-04-25T03:24:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59850K .......... .......... .......... .......... .......... 35% 25.6M 4s 2024-04-25T03:24:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59900K .......... .......... .......... .......... .......... 35% 42.6M 4s 2024-04-25T03:24:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 59950K .......... .......... .......... .......... .......... 35% 22.4M 4s 2024-04-25T03:24:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60000K .......... .......... .......... .......... .......... 35% 22.1M 4s 2024-04-25T03:24:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60050K .......... .......... .......... .......... .......... 35% 67.8M 4s 2024-04-25T03:24:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60100K .......... .......... .......... .......... .......... 35% 23.8M 4s 2024-04-25T03:24:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60150K .......... .......... .......... .......... .......... 35% 32.1M 4s 2024-04-25T03:24:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60200K .......... .......... .......... .......... .......... 35% 24.0M 4s 2024-04-25T03:24:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60250K .......... .......... .......... .......... .......... 35% 27.5M 4s 2024-04-25T03:24:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60300K .......... .......... .......... .......... .......... 35% 46.3M 4s 2024-04-25T03:24:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60350K .......... .......... .......... .......... .......... 35% 21.6M 4s 2024-04-25T03:24:18.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60400K .......... .......... .......... .......... .......... 35% 38.0M 4s 2024-04-25T03:24:18.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60450K .......... .......... .......... .......... .......... 35% 28.5M 4s 2024-04-25T03:24:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60500K .......... .......... .......... .......... .......... 35% 23.7M 4s 2024-04-25T03:24:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60550K .......... .......... .......... .......... .......... 35% 41.8M 4s 2024-04-25T03:24:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60600K .......... .......... .......... .......... .......... 35% 26.0M 4s 2024-04-25T03:24:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60650K .......... .......... .......... .......... .......... 36% 33.0M 4s 2024-04-25T03:24:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60700K .......... .......... .......... .......... .......... 36% 25.7M 4s 2024-04-25T03:24:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60750K .......... .......... .......... .......... .......... 36% 22.2M 4s 2024-04-25T03:24:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60800K .......... .......... .......... .......... .......... 36% 24.4M 4s 2024-04-25T03:24:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60850K .......... .......... .......... .......... .......... 36% 50.1M 4s 2024-04-25T03:24:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60900K .......... .......... .......... .......... .......... 36% 29.0M 4s 2024-04-25T03:24:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 60950K .......... .......... .......... .......... .......... 36% 29.2M 4s 2024-04-25T03:24:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61000K .......... .......... .......... .......... .......... 36% 25.9M 4s 2024-04-25T03:24:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61050K .......... .......... .......... .......... .......... 36% 43.1M 4s 2024-04-25T03:24:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61100K .......... .......... .......... .......... .......... 36% 23.6M 4s 2024-04-25T03:24:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61150K .......... .......... .......... .......... .......... 36% 25.5M 4s 2024-04-25T03:24:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61200K .......... .......... .......... .......... .......... 36% 41.4M 4s 2024-04-25T03:24:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61250K .......... .......... .......... .......... .......... 36% 23.7M 4s 2024-04-25T03:24:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61300K .......... .......... .......... .......... .......... 36% 24.1M 4s 2024-04-25T03:24:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61350K .......... .......... .......... .......... .......... 36% 37.8M 4s 2024-04-25T03:24:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61400K .......... .......... .......... .......... .......... 36% 23.7M 4s 2024-04-25T03:24:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61450K .......... .......... .......... .......... .......... 36% 55.3M 4s 2024-04-25T03:24:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61500K .......... .......... .......... .......... .......... 36% 22.0M 4s 2024-04-25T03:24:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61550K .......... .......... .......... .......... .......... 36% 24.0M 4s 2024-04-25T03:24:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61600K .......... .......... .......... .......... .......... 36% 40.2M 4s 2024-04-25T03:24:18.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61650K .......... .......... .......... .......... .......... 36% 27.2M 4s 2024-04-25T03:24:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61700K .......... .......... .......... .......... .......... 36% 38.6M 4s 2024-04-25T03:24:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61750K .......... .......... .......... .......... .......... 36% 22.7M 4s 2024-04-25T03:24:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61800K .......... .......... .......... .......... .......... 36% 54.3M 4s 2024-04-25T03:24:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61850K .......... .......... .......... .......... .......... 36% 22.5M 4s 2024-04-25T03:24:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61900K .......... .......... .......... .......... .......... 36% 31.4M 4s 2024-04-25T03:24:18.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 61950K .......... .......... .......... .......... .......... 36% 19.2M 4s 2024-04-25T03:24:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62000K .......... .......... .......... .......... .......... 36% 37.4M 4s 2024-04-25T03:24:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62050K .......... .......... .......... .......... .......... 36% 23.6M 4s 2024-04-25T03:24:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62100K .......... .......... .......... .......... .......... 36% 43.0M 4s 2024-04-25T03:24:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62150K .......... .......... .......... .......... .......... 36% 26.2M 4s 2024-04-25T03:24:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62200K .......... .......... .......... .......... .......... 36% 36.9M 4s 2024-04-25T03:24:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62250K .......... .......... .......... .......... .......... 36% 24.2M 4s 2024-04-25T03:24:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62300K .......... .......... .......... .......... .......... 36% 50.9M 4s 2024-04-25T03:24:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62350K .......... .......... .......... .......... .......... 37% 23.1M 4s 2024-04-25T03:24:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62400K .......... .......... .......... .......... .......... 37% 23.8M 4s 2024-04-25T03:24:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62450K .......... .......... .......... .......... .......... 37% 34.2M 4s 2024-04-25T03:24:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62500K .......... .......... .......... .......... .......... 37% 24.0M 4s 2024-04-25T03:24:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62550K .......... .......... .......... .......... .......... 37% 23.6M 4s 2024-04-25T03:24:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62600K .......... .......... .......... .......... .......... 37% 45.4M 4s 2024-04-25T03:24:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62650K .......... .......... .......... .......... .......... 37% 24.3M 4s 2024-04-25T03:24:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62700K .......... .......... .......... .......... .......... 37% 36.8M 4s 2024-04-25T03:24:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62750K .......... .......... .......... .......... .......... 37% 25.6M 4s 2024-04-25T03:24:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62800K .......... .......... .......... .......... .......... 37% 22.5M 4s 2024-04-25T03:24:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62850K .......... .......... .......... .......... .......... 37% 46.9M 4s 2024-04-25T03:24:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62900K .......... .......... .......... .......... .......... 37% 27.4M 4s 2024-04-25T03:24:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 62950K .......... .......... .......... .......... .......... 37% 35.4M 4s 2024-04-25T03:24:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63000K .......... .......... .......... .......... .......... 37% 21.7M 4s 2024-04-25T03:24:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63050K .......... .......... .......... .......... .......... 37% 27.3M 4s 2024-04-25T03:24:18.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63100K .......... .......... .......... .......... .......... 37% 54.6M 4s 2024-04-25T03:24:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63150K .......... .......... .......... .......... .......... 37% 22.8M 4s 2024-04-25T03:24:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63200K .......... .......... .......... .......... .......... 37% 23.8M 4s 2024-04-25T03:24:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63250K .......... .......... .......... .......... .......... 37% 41.2M 4s 2024-04-25T03:24:18.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63300K .......... .......... .......... .......... .......... 37% 21.4M 4s 2024-04-25T03:24:18.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63350K .......... .......... .......... .......... .......... 37% 37.3M 4s 2024-04-25T03:24:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63400K .......... .......... .......... .......... .......... 37% 26.8M 4s 2024-04-25T03:24:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63450K .......... .......... .......... .......... .......... 37% 37.1M 4s 2024-04-25T03:24:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63500K .......... .......... .......... .......... .......... 37% 23.2M 4s 2024-04-25T03:24:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63550K .......... .......... .......... .......... .......... 37% 20.3M 4s 2024-04-25T03:24:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63600K .......... .......... .......... .......... .......... 37% 27.3M 4s 2024-04-25T03:24:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63650K .......... .......... .......... .......... .......... 37% 59.1M 4s 2024-04-25T03:24:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63700K .......... .......... .......... .......... .......... 37% 23.5M 4s 2024-04-25T03:24:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63750K .......... .......... .......... .......... .......... 37% 24.5M 4s 2024-04-25T03:24:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63800K .......... .......... .......... .......... .......... 37% 34.0M 4s 2024-04-25T03:24:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63850K .......... .......... .......... .......... .......... 37% 50.3M 4s 2024-04-25T03:24:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63900K .......... .......... .......... .......... .......... 37% 28.2M 4s 2024-04-25T03:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 63950K .......... .......... .......... .......... .......... 37% 19.7M 4s 2024-04-25T03:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64000K .......... .......... .......... .......... .......... 37% 27.0M 4s 2024-04-25T03:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64050K .......... .......... .......... .......... .......... 38% 54.0M 4s 2024-04-25T03:24:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64100K .......... .......... .......... .......... .......... 38% 29.3M 4s 2024-04-25T03:24:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64150K .......... .......... .......... .......... .......... 38% 23.7M 4s 2024-04-25T03:24:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64200K .......... .......... .......... .......... .......... 38% 42.2M 4s 2024-04-25T03:24:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64250K .......... .......... .......... .......... .......... 38% 30.3M 4s 2024-04-25T03:24:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64300K .......... .......... .......... .......... .......... 38% 22.1M 4s 2024-04-25T03:24:18.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64350K .......... .......... .......... .......... .......... 38% 32.4M 4s 2024-04-25T03:24:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64400K .......... .......... .......... .......... .......... 38% 28.9M 4s 2024-04-25T03:24:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64450K .......... .......... .......... .......... .......... 38% 22.9M 4s 2024-04-25T03:24:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64500K .......... .......... .......... .......... .......... 38% 24.1M 4s 2024-04-25T03:24:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64550K .......... .......... .......... .......... .......... 38% 26.8M 4s 2024-04-25T03:24:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64600K .......... .......... .......... .......... .......... 38% 144M 4s 2024-04-25T03:24:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64650K .......... .......... .......... .......... .......... 38% 21.2M 4s 2024-04-25T03:24:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64700K .......... .......... .......... .......... .......... 38% 33.8M 4s 2024-04-25T03:24:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64750K .......... .......... .......... .......... .......... 38% 24.8M 4s 2024-04-25T03:24:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64800K .......... .......... .......... .......... .......... 38% 24.9M 4s 2024-04-25T03:24:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64850K .......... .......... .......... .......... .......... 38% 40.8M 4s 2024-04-25T03:24:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64900K .......... .......... .......... .......... .......... 38% 30.2M 4s 2024-04-25T03:24:18.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 64950K .......... .......... .......... .......... .......... 38% 21.8M 4s 2024-04-25T03:24:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65000K .......... .......... .......... .......... .......... 38% 24.0M 4s 2024-04-25T03:24:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65050K .......... .......... .......... .......... .......... 38% 40.6M 4s 2024-04-25T03:24:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65100K .......... .......... .......... .......... .......... 38% 50.7M 4s 2024-04-25T03:24:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65150K .......... .......... .......... .......... .......... 38% 21.9M 4s 2024-04-25T03:24:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65200K .......... .......... .......... .......... .......... 38% 25.4M 4s 2024-04-25T03:24:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65250K .......... .......... .......... .......... .......... 38% 39.1M 4s 2024-04-25T03:24:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65300K .......... .......... .......... .......... .......... 38% 21.9M 4s 2024-04-25T03:24:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65350K .......... .......... .......... .......... .......... 38% 26.2M 4s 2024-04-25T03:24:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65400K .......... .......... .......... .......... .......... 38% 44.6M 4s 2024-04-25T03:24:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65450K .......... .......... .......... .......... .......... 38% 26.2M 4s 2024-04-25T03:24:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65500K .......... .......... .......... .......... .......... 38% 38.4M 4s 2024-04-25T03:24:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65550K .......... .......... .......... .......... .......... 38% 23.1M 4s 2024-04-25T03:24:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65600K .......... .......... .......... .......... .......... 38% 25.1M 4s 2024-04-25T03:24:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65650K .......... .......... .......... .......... .......... 38% 52.6M 4s 2024-04-25T03:24:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65700K .......... .......... .......... .......... .......... 39% 21.9M 4s 2024-04-25T03:24:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65750K .......... .......... .......... .......... .......... 39% 38.5M 4s 2024-04-25T03:24:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65800K .......... .......... .......... .......... .......... 39% 24.2M 4s 2024-04-25T03:24:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65850K .......... .......... .......... .......... .......... 39% 23.2M 4s 2024-04-25T03:24:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65900K .......... .......... .......... .......... .......... 39% 46.4M 4s 2024-04-25T03:24:18.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 65950K .......... .......... .......... .......... .......... 39% 21.5M 4s 2024-04-25T03:24:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66000K .......... .......... .......... .......... .......... 39% 28.4M 4s 2024-04-25T03:24:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66050K .......... .......... .......... .......... .......... 39% 45.0M 4s 2024-04-25T03:24:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66100K .......... .......... .......... .......... .......... 39% 22.5M 4s 2024-04-25T03:24:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66150K .......... .......... .......... .......... .......... 39% 48.4M 4s 2024-04-25T03:24:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66200K .......... .......... .......... .......... .......... 39% 23.8M 4s 2024-04-25T03:24:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66250K .......... .......... .......... .......... .......... 39% 32.0M 4s 2024-04-25T03:24:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66300K .......... .......... .......... .......... .......... 39% 27.9M 4s 2024-04-25T03:24:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66350K .......... .......... .......... .......... .......... 39% 24.3M 4s 2024-04-25T03:24:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66400K .......... .......... .......... .......... .......... 39% 22.8M 4s 2024-04-25T03:24:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66450K .......... .......... .......... .......... .......... 39% 53.2M 4s 2024-04-25T03:24:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66500K .......... .......... .......... .......... .......... 39% 17.3M 4s 2024-04-25T03:24:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66550K .......... .......... .......... .......... .......... 39% 37.9M 4s 2024-04-25T03:24:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66600K .......... .......... .......... .......... .......... 39% 23.2M 4s 2024-04-25T03:24:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66650K .......... .......... .......... .......... .......... 39% 27.2M 4s 2024-04-25T03:24:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66700K .......... .......... .......... .......... .......... 39% 20.9M 4s 2024-04-25T03:24:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66750K .......... .......... .......... .......... .......... 39% 8.50M 4s 2024-04-25T03:24:18.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66800K .......... .......... .......... .......... .......... 39% 16.3M 4s 2024-04-25T03:24:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66850K .......... .......... .......... .......... .......... 39% 28.1M 4s 2024-04-25T03:24:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66900K .......... .......... .......... .......... .......... 39% 41.5M 4s 2024-04-25T03:24:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 66950K .......... .......... .......... .......... .......... 39% 16.3M 4s 2024-04-25T03:24:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67000K .......... .......... .......... .......... .......... 39% 30.9M 4s 2024-04-25T03:24:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67050K .......... .......... .......... .......... .......... 39% 23.5M 4s 2024-04-25T03:24:18.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67100K .......... .......... .......... .......... .......... 39% 27.6M 4s 2024-04-25T03:24:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67150K .......... .......... .......... .......... .......... 39% 23.8M 4s 2024-04-25T03:24:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67200K .......... .......... .......... .......... .......... 39% 44.1M 4s 2024-04-25T03:24:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67250K .......... .......... .......... .......... .......... 39% 25.0M 4s 2024-04-25T03:24:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67300K .......... .......... .......... .......... .......... 39% 47.0M 4s 2024-04-25T03:24:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67350K .......... .......... .......... .......... .......... 39% 20.8M 4s 2024-04-25T03:24:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67400K .......... .......... .......... .......... .......... 40% 42.0M 4s 2024-04-25T03:24:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67450K .......... .......... .......... .......... .......... 40% 25.9M 4s 2024-04-25T03:24:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67500K .......... .......... .......... .......... .......... 40% 43.6M 4s 2024-04-25T03:24:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67550K .......... .......... .......... .......... .......... 40% 16.4M 4s 2024-04-25T03:24:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67600K .......... .......... .......... .......... .......... 40% 35.8M 4s 2024-04-25T03:24:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67650K .......... .......... .......... .......... .......... 40% 24.7M 4s 2024-04-25T03:24:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67700K .......... .......... .......... .......... .......... 40% 38.0M 4s 2024-04-25T03:24:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67750K .......... .......... .......... .......... .......... 40% 28.2M 4s 2024-04-25T03:24:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67800K .......... .......... .......... .......... .......... 40% 35.7M 4s 2024-04-25T03:24:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67850K .......... .......... .......... .......... .......... 40% 22.8M 4s 2024-04-25T03:24:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67900K .......... .......... .......... .......... .......... 40% 57.5M 4s 2024-04-25T03:24:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 67950K .......... .......... .......... .......... .......... 40% 23.8M 4s 2024-04-25T03:24:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68000K .......... .......... .......... .......... .......... 40% 24.8M 4s 2024-04-25T03:24:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68050K .......... .......... .......... .......... .......... 40% 29.8M 4s 2024-04-25T03:24:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68100K .......... .......... .......... .......... .......... 40% 28.4M 4s 2024-04-25T03:24:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68150K .......... .......... .......... .......... .......... 40% 25.9M 4s 2024-04-25T03:24:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68200K .......... .......... .......... .......... .......... 40% 41.7M 4s 2024-04-25T03:24:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68250K .......... .......... .......... .......... .......... 40% 21.1M 4s 2024-04-25T03:24:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68300K .......... .......... .......... .......... .......... 40% 40.2M 4s 2024-04-25T03:24:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68350K .......... .......... .......... .......... .......... 40% 22.1M 4s 2024-04-25T03:24:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68400K .......... .......... .......... .......... .......... 40% 29.0M 4s 2024-04-25T03:24:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68450K .......... .......... .......... .......... .......... 40% 44.4M 4s 2024-04-25T03:24:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68500K .......... .......... .......... .......... .......... 40% 23.9M 4s 2024-04-25T03:24:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68550K .......... .......... .......... .......... .......... 40% 41.8M 4s 2024-04-25T03:24:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68600K .......... .......... .......... .......... .......... 40% 19.7M 4s 2024-04-25T03:24:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68650K .......... .......... .......... .......... .......... 40% 28.3M 4s 2024-04-25T03:24:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68700K .......... .......... .......... .......... .......... 40% 35.4M 4s 2024-04-25T03:24:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68750K .......... .......... .......... .......... .......... 40% 27.1M 4s 2024-04-25T03:24:18.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68800K .......... .......... .......... .......... .......... 40% 24.5M 4s 2024-04-25T03:24:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68850K .......... .......... .......... .......... .......... 40% 34.2M 4s 2024-04-25T03:24:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68900K .......... .......... .......... .......... .......... 40% 14.4M 4s 2024-04-25T03:24:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 68950K .......... .......... .......... .......... .......... 40% 42.7M 4s 2024-04-25T03:24:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69000K .......... .......... .......... .......... .......... 40% 23.2M 4s 2024-04-25T03:24:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69050K .......... .......... .......... .......... .......... 40% 45.2M 4s 2024-04-25T03:24:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69100K .......... .......... .......... .......... .......... 41% 15.1M 4s 2024-04-25T03:24:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69150K .......... .......... .......... .......... .......... 41% 23.9M 4s 2024-04-25T03:24:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69200K .......... .......... .......... .......... .......... 41% 23.8M 4s 2024-04-25T03:24:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69250K .......... .......... .......... .......... .......... 41% 44.0M 4s 2024-04-25T03:24:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69300K .......... .......... .......... .......... .......... 41% 25.0M 4s 2024-04-25T03:24:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69350K .......... .......... .......... .......... .......... 41% 36.4M 4s 2024-04-25T03:24:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69400K .......... .......... .......... .......... .......... 41% 22.4M 4s 2024-04-25T03:24:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69450K .......... .......... .......... .......... .......... 41% 48.2M 4s 2024-04-25T03:24:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69500K .......... .......... .......... .......... .......... 41% 25.1M 4s 2024-04-25T03:24:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69550K .......... .......... .......... .......... .......... 41% 29.8M 4s 2024-04-25T03:24:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69600K .......... .......... .......... .......... .......... 41% 28.3M 4s 2024-04-25T03:24:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69650K .......... .......... .......... .......... .......... 41% 25.4M 4s 2024-04-25T03:24:19.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69700K .......... .......... .......... .......... .......... 41% 45.3M 4s 2024-04-25T03:24:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69750K .......... .......... .......... .......... .......... 41% 20.3M 4s 2024-04-25T03:24:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69800K .......... .......... .......... .......... .......... 41% 29.1M 4s 2024-04-25T03:24:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69850K .......... .......... .......... .......... .......... 41% 36.6M 4s 2024-04-25T03:24:19.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69900K .......... .......... .......... .......... .......... 41% 25.4M 4s 2024-04-25T03:24:19.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 69950K .......... .......... .......... .......... .......... 41% 23.4M 4s 2024-04-25T03:24:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70000K .......... .......... .......... .......... .......... 41% 47.3M 4s 2024-04-25T03:24:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70050K .......... .......... .......... .......... .......... 41% 22.8M 4s 2024-04-25T03:24:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70100K .......... .......... .......... .......... .......... 41% 41.3M 4s 2024-04-25T03:24:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70150K .......... .......... .......... .......... .......... 41% 22.8M 4s 2024-04-25T03:24:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70200K .......... .......... .......... .......... .......... 41% 54.1M 4s 2024-04-25T03:24:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70250K .......... .......... .......... .......... .......... 41% 22.5M 4s 2024-04-25T03:24:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70300K .......... .......... .......... .......... .......... 41% 36.8M 4s 2024-04-25T03:24:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70350K .......... .......... .......... .......... .......... 41% 23.5M 4s 2024-04-25T03:24:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70400K .......... .......... .......... .......... .......... 41% 22.4M 4s 2024-04-25T03:24:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70450K .......... .......... .......... .......... .......... 41% 30.0M 4s 2024-04-25T03:24:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70500K .......... .......... .......... .......... .......... 41% 44.0M 4s 2024-04-25T03:24:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70550K .......... .......... .......... .......... .......... 41% 24.4M 4s 2024-04-25T03:24:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70600K .......... .......... .......... .......... .......... 41% 36.3M 4s 2024-04-25T03:24:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70650K .......... .......... .......... .......... .......... 41% 20.5M 4s 2024-04-25T03:24:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70700K .......... .......... .......... .......... .......... 41% 58.6M 4s 2024-04-25T03:24:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70750K .......... .......... .......... .......... .......... 42% 20.0M 4s 2024-04-25T03:24:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70800K .......... .......... .......... .......... .......... 42% 29.0M 4s 2024-04-25T03:24:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70850K .......... .......... .......... .......... .......... 42% 32.6M 4s 2024-04-25T03:24:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70900K .......... .......... .......... .......... .......... 42% 28.4M 4s 2024-04-25T03:24:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 70950K .......... .......... .......... .......... .......... 42% 24.7M 4s 2024-04-25T03:24:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71000K .......... .......... .......... .......... .......... 42% 37.5M 4s 2024-04-25T03:24:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71050K .......... .......... .......... .......... .......... 42% 29.2M 4s 2024-04-25T03:24:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71100K .......... .......... .......... .......... .......... 42% 28.5M 4s 2024-04-25T03:24:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71150K .......... .......... .......... .......... .......... 42% 26.2M 4s 2024-04-25T03:24:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71200K .......... .......... .......... .......... .......... 42% 22.6M 4s 2024-04-25T03:24:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71250K .......... .......... .......... .......... .......... 42% 41.5M 4s 2024-04-25T03:24:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71300K .......... .......... .......... .......... .......... 42% 23.7M 4s 2024-04-25T03:24:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71350K .......... .......... .......... .......... .......... 42% 43.6M 4s 2024-04-25T03:24:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71400K .......... .......... .......... .......... .......... 42% 23.1M 4s 2024-04-25T03:24:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71450K .......... .......... .......... .......... .......... 42% 48.2M 4s 2024-04-25T03:24:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71500K .......... .......... .......... .......... .......... 42% 21.4M 4s 2024-04-25T03:24:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71550K .......... .......... .......... .......... .......... 42% 23.9M 4s 2024-04-25T03:24:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71600K .......... .......... .......... .......... .......... 42% 24.4M 4s 2024-04-25T03:24:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71650K .......... .......... .......... .......... .......... 42% 41.3M 4s 2024-04-25T03:24:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71700K .......... .......... .......... .......... .......... 42% 16.0M 4s 2024-04-25T03:24:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71750K .......... .......... .......... .......... .......... 42% 30.4M 4s 2024-04-25T03:24:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71800K .......... .......... .......... .......... .......... 42% 22.0M 4s 2024-04-25T03:24:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71850K .......... .......... .......... .......... .......... 42% 39.0M 4s 2024-04-25T03:24:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71900K .......... .......... .......... .......... .......... 42% 26.1M 4s 2024-04-25T03:24:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 71950K .......... .......... .......... .......... .......... 42% 24.9M 4s 2024-04-25T03:24:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72000K .......... .......... .......... .......... .......... 42% 46.3M 4s 2024-04-25T03:24:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72050K .......... .......... .......... .......... .......... 42% 22.6M 4s 2024-04-25T03:24:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72100K .......... .......... .......... .......... .......... 42% 24.0M 4s 2024-04-25T03:24:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72150K .......... .......... .......... .......... .......... 42% 33.0M 4s 2024-04-25T03:24:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72200K .......... .......... .......... .......... .......... 42% 25.9M 4s 2024-04-25T03:24:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72250K .......... .......... .......... .......... .......... 42% 41.2M 4s 2024-04-25T03:24:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72300K .......... .......... .......... .......... .......... 42% 24.8M 4s 2024-04-25T03:24:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72350K .......... .......... .......... .......... .......... 42% 23.7M 4s 2024-04-25T03:24:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72400K .......... .......... .......... .......... .......... 42% 24.9M 4s 2024-04-25T03:24:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72450K .......... .......... .......... .......... .......... 43% 36.8M 4s 2024-04-25T03:24:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72500K .......... .......... .......... .......... .......... 43% 38.5M 4s 2024-04-25T03:24:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72550K .......... .......... .......... .......... .......... 43% 22.3M 4s 2024-04-25T03:24:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72600K .......... .......... .......... .......... .......... 43% 44.5M 4s 2024-04-25T03:24:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72650K .......... .......... .......... .......... .......... 43% 24.4M 4s 2024-04-25T03:24:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72700K .......... .......... .......... .......... .......... 43% 23.1M 4s 2024-04-25T03:24:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72750K .......... .......... .......... .......... .......... 43% 25.1M 4s 2024-04-25T03:24:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72800K .......... .......... .......... .......... .......... 43% 44.0M 4s 2024-04-25T03:24:19.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72850K .......... .......... .......... .......... .......... 43% 25.5M 4s 2024-04-25T03:24:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72900K .......... .......... .......... .......... .......... 43% 39.0M 4s 2024-04-25T03:24:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 72950K .......... .......... .......... .......... .......... 43% 29.9M 4s 2024-04-25T03:24:19.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73000K .......... .......... .......... .......... .......... 43% 33.1M 4s 2024-04-25T03:24:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73050K .......... .......... .......... .......... .......... 43% 22.4M 4s 2024-04-25T03:24:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73100K .......... .......... .......... .......... .......... 43% 27.3M 4s 2024-04-25T03:24:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73150K .......... .......... .......... .......... .......... 43% 38.2M 4s 2024-04-25T03:24:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73200K .......... .......... .......... .......... .......... 43% 22.3M 4s 2024-04-25T03:24:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73250K .......... .......... .......... .......... .......... 43% 39.3M 4s 2024-04-25T03:24:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73300K .......... .......... .......... .......... .......... 43% 29.2M 4s 2024-04-25T03:24:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73350K .......... .......... .......... .......... .......... 43% 30.8M 4s 2024-04-25T03:24:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73400K .......... .......... .......... .......... .......... 43% 22.4M 4s 2024-04-25T03:24:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73450K .......... .......... .......... .......... .......... 43% 29.3M 4s 2024-04-25T03:24:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73500K .......... .......... .......... .......... .......... 43% 42.8M 4s 2024-04-25T03:24:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73550K .......... .......... .......... .......... .......... 43% 21.3M 4s 2024-04-25T03:24:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73600K .......... .......... .......... .......... .......... 43% 25.6M 4s 2024-04-25T03:24:19.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73650K .......... .......... .......... .......... .......... 43% 37.9M 4s 2024-04-25T03:24:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73700K .......... .......... .......... .......... .......... 43% 24.7M 4s 2024-04-25T03:24:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73750K .......... .......... .......... .......... .......... 43% 22.8M 4s 2024-04-25T03:24:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73800K .......... .......... .......... .......... .......... 43% 46.1M 4s 2024-04-25T03:24:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73850K .......... .......... .......... .......... .......... 43% 23.9M 3s 2024-04-25T03:24:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73900K .......... .......... .......... .......... .......... 43% 33.6M 3s 2024-04-25T03:24:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 73950K .......... .......... .......... .......... .......... 43% 23.6M 3s 2024-04-25T03:24:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74000K .......... .......... .......... .......... .......... 43% 28.0M 3s 2024-04-25T03:24:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74050K .......... .......... .......... .......... .......... 43% 40.7M 3s 2024-04-25T03:24:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74100K .......... .......... .......... .......... .......... 43% 25.2M 3s 2024-04-25T03:24:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74150K .......... .......... .......... .......... .......... 44% 39.5M 3s 2024-04-25T03:24:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74200K .......... .......... .......... .......... .......... 44% 24.5M 3s 2024-04-25T03:24:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74250K .......... .......... .......... .......... .......... 44% 41.4M 3s 2024-04-25T03:24:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74300K .......... .......... .......... .......... .......... 44% 25.8M 3s 2024-04-25T03:24:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74350K .......... .......... .......... .......... .......... 44% 22.7M 3s 2024-04-25T03:24:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74400K .......... .......... .......... .......... .......... 44% 23.8M 3s 2024-04-25T03:24:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74450K .......... .......... .......... .......... .......... 44% 45.3M 3s 2024-04-25T03:24:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74500K .......... .......... .......... .......... .......... 44% 24.1M 3s 2024-04-25T03:24:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74550K .......... .......... .......... .......... .......... 44% 40.7M 3s 2024-04-25T03:24:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74600K .......... .......... .......... .......... .......... 44% 23.0M 3s 2024-04-25T03:24:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74650K .......... .......... .......... .......... .......... 44% 39.7M 3s 2024-04-25T03:24:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74700K .......... .......... .......... .......... .......... 44% 25.6M 3s 2024-04-25T03:24:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74750K .......... .......... .......... .......... .......... 44% 23.2M 3s 2024-04-25T03:24:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74800K .......... .......... .......... .......... .......... 44% 50.3M 3s 2024-04-25T03:24:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74850K .......... .......... .......... .......... .......... 44% 26.9M 3s 2024-04-25T03:24:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74900K .......... .......... .......... .......... .......... 44% 32.1M 3s 2024-04-25T03:24:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 74950K .......... .......... .......... .......... .......... 44% 23.7M 3s 2024-04-25T03:24:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75000K .......... .......... .......... .......... .......... 44% 25.2M 3s 2024-04-25T03:24:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75050K .......... .......... .......... .......... .......... 44% 38.7M 3s 2024-04-25T03:24:19.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75100K .......... .......... .......... .......... .......... 44% 24.5M 3s 2024-04-25T03:24:19.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75150K .......... .......... .......... .......... .......... 44% 25.5M 3s 2024-04-25T03:24:19.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75200K .......... .......... .......... .......... .......... 44% 36.1M 3s 2024-04-25T03:24:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75250K .......... .......... .......... .......... .......... 44% 25.4M 3s 2024-04-25T03:24:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75300K .......... .......... .......... .......... .......... 44% 51.7M 3s 2024-04-25T03:24:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75350K .......... .......... .......... .......... .......... 44% 24.6M 3s 2024-04-25T03:24:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75400K .......... .......... .......... .......... .......... 44% 35.9M 3s 2024-04-25T03:24:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75450K .......... .......... .......... .......... .......... 44% 22.9M 3s 2024-04-25T03:24:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75500K .......... .......... .......... .......... .......... 44% 23.0M 3s 2024-04-25T03:24:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75550K .......... .......... .......... .......... .......... 44% 23.9M 3s 2024-04-25T03:24:19.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75600K .......... .......... .......... .......... .......... 44% 52.4M 3s 2024-04-25T03:24:19.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75650K .......... .......... .......... .......... .......... 44% 25.7M 3s 2024-04-25T03:24:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75700K .......... .......... .......... .......... .......... 44% 35.9M 3s 2024-04-25T03:24:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75750K .......... .......... .......... .......... .......... 44% 22.0M 3s 2024-04-25T03:24:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75800K .......... .......... .......... .......... .......... 44% 59.1M 3s 2024-04-25T03:24:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75850K .......... .......... .......... .......... .......... 45% 21.6M 3s 2024-04-25T03:24:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75900K .......... .......... .......... .......... .......... 45% 40.0M 3s 2024-04-25T03:24:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 75950K .......... .......... .......... .......... .......... 45% 23.3M 3s 2024-04-25T03:24:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76000K .......... .......... .......... .......... .......... 45% 22.1M 3s 2024-04-25T03:24:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76050K .......... .......... .......... .......... .......... 45% 22.9M 3s 2024-04-25T03:24:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76100K .......... .......... .......... .......... .......... 45% 51.2M 3s 2024-04-25T03:24:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76150K .......... .......... .......... .......... .......... 45% 20.6M 3s 2024-04-25T03:24:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76200K .......... .......... .......... .......... .......... 45% 38.1M 3s 2024-04-25T03:24:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76250K .......... .......... .......... .......... .......... 45% 23.9M 3s 2024-04-25T03:24:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76300K .......... .......... .......... .......... .......... 45% 43.1M 3s 2024-04-25T03:24:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76350K .......... .......... .......... .......... .......... 45% 22.8M 3s 2024-04-25T03:24:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76400K .......... .......... .......... .......... .......... 45% 27.9M 3s 2024-04-25T03:24:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76450K .......... .......... .......... .......... .......... 45% 28.9M 3s 2024-04-25T03:24:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76500K .......... .......... .......... .......... .......... 45% 27.5M 3s 2024-04-25T03:24:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76550K .......... .......... .......... .......... .......... 45% 48.5M 3s 2024-04-25T03:24:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76600K .......... .......... .......... .......... .......... 45% 23.7M 3s 2024-04-25T03:24:19.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76650K .......... .......... .......... .......... .......... 45% 32.9M 3s 2024-04-25T03:24:19.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76700K .......... .......... .......... .......... .......... 45% 20.8M 3s 2024-04-25T03:24:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76750K .......... .......... .......... .......... .......... 45% 27.6M 3s 2024-04-25T03:24:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76800K .......... .......... .......... .......... .......... 45% 21.2M 3s 2024-04-25T03:24:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76850K .......... .......... .......... .......... .......... 45% 45.8M 3s 2024-04-25T03:24:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76900K .......... .......... .......... .......... .......... 45% 14.2M 3s 2024-04-25T03:24:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 76950K .......... .......... .......... .......... .......... 45% 38.1M 3s 2024-04-25T03:24:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77000K .......... .......... .......... .......... .......... 45% 24.9M 3s 2024-04-25T03:24:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77050K .......... .......... .......... .......... .......... 45% 49.2M 3s 2024-04-25T03:24:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77100K .......... .......... .......... .......... .......... 45% 24.8M 3s 2024-04-25T03:24:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77150K .......... .......... .......... .......... .......... 45% 23.0M 3s 2024-04-25T03:24:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77200K .......... .......... .......... .......... .......... 45% 33.4M 3s 2024-04-25T03:24:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77250K .......... .......... .......... .......... .......... 45% 24.4M 3s 2024-04-25T03:24:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77300K .......... .......... .......... .......... .......... 45% 24.2M 3s 2024-04-25T03:24:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77350K .......... .......... .......... .......... .......... 45% 39.9M 3s 2024-04-25T03:24:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77400K .......... .......... .......... .......... .......... 45% 28.6M 3s 2024-04-25T03:24:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77450K .......... .......... .......... .......... .......... 45% 37.9M 3s 2024-04-25T03:24:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77500K .......... .......... .......... .......... .......... 46% 24.6M 3s 2024-04-25T03:24:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77550K .......... .......... .......... .......... .......... 46% 23.9M 3s 2024-04-25T03:24:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77600K .......... .......... .......... .......... .......... 46% 44.6M 3s 2024-04-25T03:24:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77650K .......... .......... .......... .......... .......... 46% 24.5M 3s 2024-04-25T03:24:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77700K .......... .......... .......... .......... .......... 46% 38.6M 3s 2024-04-25T03:24:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77750K .......... .......... .......... .......... .......... 46% 29.5M 3s 2024-04-25T03:24:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77800K .......... .......... .......... .......... .......... 46% 18.7M 3s 2024-04-25T03:24:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77850K .......... .......... .......... .......... .......... 46% 74.2M 3s 2024-04-25T03:24:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77900K .......... .......... .......... .......... .......... 46% 27.1M 3s 2024-04-25T03:24:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 77950K .......... .......... .......... .......... .......... 46% 20.8M 3s 2024-04-25T03:24:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78000K .......... .......... .......... .......... .......... 46% 38.1M 3s 2024-04-25T03:24:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78050K .......... .......... .......... .......... .......... 46% 24.9M 3s 2024-04-25T03:24:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78100K .......... .......... .......... .......... .......... 46% 45.1M 3s 2024-04-25T03:24:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78150K .......... .......... .......... .......... .......... 46% 22.7M 3s 2024-04-25T03:24:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78200K .......... .......... .......... .......... .......... 46% 39.8M 3s 2024-04-25T03:24:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78250K .......... .......... .......... .......... .......... 46% 24.2M 3s 2024-04-25T03:24:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78300K .......... .......... .......... .......... .......... 46% 25.2M 3s 2024-04-25T03:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78350K .......... .......... .......... .......... .......... 46% 21.7M 3s 2024-04-25T03:24:19.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78400K .......... .......... .......... .......... .......... 46% 61.4M 3s 2024-04-25T03:24:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78450K .......... .......... .......... .......... .......... 46% 22.2M 3s 2024-04-25T03:24:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78500K .......... .......... .......... .......... .......... 46% 42.1M 3s 2024-04-25T03:24:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78550K .......... .......... .......... .......... .......... 46% 21.0M 3s 2024-04-25T03:24:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78600K .......... .......... .......... .......... .......... 46% 40.8M 3s 2024-04-25T03:24:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78650K .......... .......... .......... .......... .......... 46% 28.5M 3s 2024-04-25T03:24:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78700K .......... .......... .......... .......... .......... 46% 41.2M 3s 2024-04-25T03:24:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78750K .......... .......... .......... .......... .......... 46% 19.0M 3s 2024-04-25T03:24:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78800K .......... .......... .......... .......... .......... 46% 25.3M 3s 2024-04-25T03:24:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78850K .......... .......... .......... .......... .......... 46% 25.5M 3s 2024-04-25T03:24:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78900K .......... .......... .......... .......... .......... 46% 45.7M 3s 2024-04-25T03:24:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 78950K .......... .......... .......... .......... .......... 46% 22.9M 3s 2024-04-25T03:24:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79000K .......... .......... .......... .......... .......... 46% 36.4M 3s 2024-04-25T03:24:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79050K .......... .......... .......... .......... .......... 46% 24.6M 3s 2024-04-25T03:24:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79100K .......... .......... .......... .......... .......... 46% 41.3M 3s 2024-04-25T03:24:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79150K .......... .......... .......... .......... .......... 46% 26.9M 3s 2024-04-25T03:24:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79200K .......... .......... .......... .......... .......... 47% 26.3M 3s 2024-04-25T03:24:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79250K .......... .......... .......... .......... .......... 47% 30.9M 3s 2024-04-25T03:24:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79300K .......... .......... .......... .......... .......... 47% 27.7M 3s 2024-04-25T03:24:19.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79350K .......... .......... .......... .......... .......... 47% 39.8M 3s 2024-04-25T03:24:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79400K .......... .......... .......... .......... .......... 47% 13.4M 3s 2024-04-25T03:24:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79450K .......... .......... .......... .......... .......... 47% 23.4M 3s 2024-04-25T03:24:19.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79500K .......... .......... .......... .......... .......... 47% 40.5M 3s 2024-04-25T03:24:19.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79550K .......... .......... .......... .......... .......... 47% 15.8M 3s 2024-04-25T03:24:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79600K .......... .......... .......... .......... .......... 47% 24.1M 3s 2024-04-25T03:24:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79650K .......... .......... .......... .......... .......... 47% 33.5M 3s 2024-04-25T03:24:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79700K .......... .......... .......... .......... .......... 47% 27.5M 3s 2024-04-25T03:24:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79750K .......... .......... .......... .......... .......... 47% 35.8M 3s 2024-04-25T03:24:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79800K .......... .......... .......... .......... .......... 47% 16.9M 3s 2024-04-25T03:24:19.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79850K .......... .......... .......... .......... .......... 47% 46.9M 3s 2024-04-25T03:24:19.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79900K .......... .......... .......... .......... .......... 47% 23.4M 3s 2024-04-25T03:24:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 79950K .......... .......... .......... .......... .......... 47% 24.7M 3s 2024-04-25T03:24:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80000K .......... .......... .......... .......... .......... 47% 9.50M 3s 2024-04-25T03:24:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80050K .......... .......... .......... .......... .......... 47% 185M 3s 2024-04-25T03:24:19.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80100K .......... .......... .......... .......... .......... 47% 32.8M 3s 2024-04-25T03:24:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80150K .......... .......... .......... .......... .......... 47% 43.6M 3s 2024-04-25T03:24:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80200K .......... .......... .......... .......... .......... 47% 23.8M 3s 2024-04-25T03:24:19.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80250K .......... .......... .......... .......... .......... 47% 38.4M 3s 2024-04-25T03:24:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80300K .......... .......... .......... .......... .......... 47% 23.5M 3s 2024-04-25T03:24:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80350K .......... .......... .......... .......... .......... 47% 25.8M 3s 2024-04-25T03:24:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80400K .......... .......... .......... .......... .......... 47% 38.1M 3s 2024-04-25T03:24:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80450K .......... .......... .......... .......... .......... 47% 26.2M 3s 2024-04-25T03:24:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80500K .......... .......... .......... .......... .......... 47% 43.9M 3s 2024-04-25T03:24:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80550K .......... .......... .......... .......... .......... 47% 22.5M 3s 2024-04-25T03:24:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80600K .......... .......... .......... .......... .......... 47% 22.2M 3s 2024-04-25T03:24:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80650K .......... .......... .......... .......... .......... 47% 57.3M 3s 2024-04-25T03:24:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80700K .......... .......... .......... .......... .......... 47% 24.5M 3s 2024-04-25T03:24:19.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80750K .......... .......... .......... .......... .......... 47% 24.0M 3s 2024-04-25T03:24:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80800K .......... .......... .......... .......... .......... 47% 35.6M 3s 2024-04-25T03:24:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80850K .......... .......... .......... .......... .......... 47% 25.1M 3s 2024-04-25T03:24:19.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80900K .......... .......... .......... .......... .......... 48% 40.8M 3s 2024-04-25T03:24:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 80950K .......... .......... .......... .......... .......... 48% 24.5M 3s 2024-04-25T03:24:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81000K .......... .......... .......... .......... .......... 48% 38.1M 3s 2024-04-25T03:24:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81050K .......... .......... .......... .......... .......... 48% 28.6M 3s 2024-04-25T03:24:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81100K .......... .......... .......... .......... .......... 48% 23.0M 3s 2024-04-25T03:24:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81150K .......... .......... .......... .......... .......... 48% 22.6M 3s 2024-04-25T03:24:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81200K .......... .......... .......... .......... .......... 48% 55.9M 3s 2024-04-25T03:24:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81250K .......... .......... .......... .......... .......... 48% 22.7M 3s 2024-04-25T03:24:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81300K .......... .......... .......... .......... .......... 48% 32.7M 3s 2024-04-25T03:24:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81350K .......... .......... .......... .......... .......... 48% 28.3M 3s 2024-04-25T03:24:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81400K .......... .......... .......... .......... .......... 48% 38.6M 3s 2024-04-25T03:24:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81450K .......... .......... .......... .......... .......... 48% 24.1M 3s 2024-04-25T03:24:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81500K .......... .......... .......... .......... .......... 48% 44.9M 3s 2024-04-25T03:24:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81550K .......... .......... .......... .......... .......... 48% 11.8M 3s 2024-04-25T03:24:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81600K .......... .......... .......... .......... .......... 48% 12.8M 3s 2024-04-25T03:24:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81650K .......... .......... .......... .......... .......... 48% 52.7M 3s 2024-04-25T03:24:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81700K .......... .......... .......... .......... .......... 48% 22.4M 3s 2024-04-25T03:24:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81750K .......... .......... .......... .......... .......... 48% 14.5M 3s 2024-04-25T03:24:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81800K .......... .......... .......... .......... .......... 48% 40.0M 3s 2024-04-25T03:24:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81850K .......... .......... .......... .......... .......... 48% 49.9M 3s 2024-04-25T03:24:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81900K .......... .......... .......... .......... .......... 48% 25.0M 3s 2024-04-25T03:24:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 81950K .......... .......... .......... .......... .......... 48% 23.6M 3s 2024-04-25T03:24:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82000K .......... .......... .......... .......... .......... 48% 23.4M 3s 2024-04-25T03:24:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82050K .......... .......... .......... .......... .......... 48% 42.8M 3s 2024-04-25T03:24:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82100K .......... .......... .......... .......... .......... 48% 36.1M 3s 2024-04-25T03:24:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82150K .......... .......... .......... .......... .......... 48% 30.1M 3s 2024-04-25T03:24:19.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82200K .......... .......... .......... .......... .......... 48% 20.8M 3s 2024-04-25T03:24:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82250K .......... .......... .......... .......... .......... 48% 29.0M 3s 2024-04-25T03:24:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82300K .......... .......... .......... .......... .......... 48% 33.5M 3s 2024-04-25T03:24:19.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82350K .......... .......... .......... .......... .......... 48% 23.6M 3s 2024-04-25T03:24:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82400K .......... .......... .......... .......... .......... 48% 25.0M 3s 2024-04-25T03:24:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82450K .......... .......... .......... .......... .......... 48% 44.1M 3s 2024-04-25T03:24:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82500K .......... .......... .......... .......... .......... 48% 23.5M 3s 2024-04-25T03:24:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82550K .......... .......... .......... .......... .......... 49% 25.9M 3s 2024-04-25T03:24:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82600K .......... .......... .......... .......... .......... 49% 34.6M 3s 2024-04-25T03:24:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82650K .......... .......... .......... .......... .......... 49% 48.4M 3s 2024-04-25T03:24:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82700K .......... .......... .......... .......... .......... 49% 19.9M 3s 2024-04-25T03:24:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82750K .......... .......... .......... .......... .......... 49% 25.5M 3s 2024-04-25T03:24:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82800K .......... .......... .......... .......... .......... 49% 24.1M 3s 2024-04-25T03:24:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82850K .......... .......... .......... .......... .......... 49% 40.4M 3s 2024-04-25T03:24:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82900K .......... .......... .......... .......... .......... 49% 24.9M 3s 2024-04-25T03:24:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 82950K .......... .......... .......... .......... .......... 49% 41.2M 3s 2024-04-25T03:24:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83000K .......... .......... .......... .......... .......... 49% 31.9M 3s 2024-04-25T03:24:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83050K .......... .......... .......... .......... .......... 49% 29.8M 3s 2024-04-25T03:24:19.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83100K .......... .......... .......... .......... .......... 49% 24.8M 3s 2024-04-25T03:24:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83150K .......... .......... .......... .......... .......... 49% 22.8M 3s 2024-04-25T03:24:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83200K .......... .......... .......... .......... .......... 49% 40.4M 3s 2024-04-25T03:24:19.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83250K .......... .......... .......... .......... .......... 49% 28.6M 3s 2024-04-25T03:24:19.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83300K .......... .......... .......... .......... .......... 49% 38.0M 3s 2024-04-25T03:24:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83350K .......... .......... .......... .......... .......... 49% 22.5M 3s 2024-04-25T03:24:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83400K .......... .......... .......... .......... .......... 49% 35.6M 3s 2024-04-25T03:24:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83450K .......... .......... .......... .......... .......... 49% 29.2M 3s 2024-04-25T03:24:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83500K .......... .......... .......... .......... .......... 49% 23.0M 3s 2024-04-25T03:24:19.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83550K .......... .......... .......... .......... .......... 49% 23.8M 3s 2024-04-25T03:24:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83600K .......... .......... .......... .......... .......... 49% 39.6M 3s 2024-04-25T03:24:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83650K .......... .......... .......... .......... .......... 49% 24.3M 3s 2024-04-25T03:24:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83700K .......... .......... .......... .......... .......... 49% 37.5M 3s 2024-04-25T03:24:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83750K .......... .......... .......... .......... .......... 49% 27.9M 3s 2024-04-25T03:24:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83800K .......... .......... .......... .......... .......... 49% 38.9M 3s 2024-04-25T03:24:19.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83850K .......... .......... .......... .......... .......... 49% 22.2M 3s 2024-04-25T03:24:19.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83900K .......... .......... .......... .......... .......... 49% 39.9M 3s 2024-04-25T03:24:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 83950K .......... .......... .......... .......... .......... 49% 17.6M 3s 2024-04-25T03:24:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84000K .......... .......... .......... .......... .......... 49% 36.1M 3s 2024-04-25T03:24:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84050K .......... .......... .......... .......... .......... 49% 27.5M 3s 2024-04-25T03:24:19.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84100K .......... .......... .......... .......... .......... 49% 41.4M 3s 2024-04-25T03:24:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84150K .......... .......... .......... .......... .......... 49% 22.9M 3s 2024-04-25T03:24:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84200K .......... .......... .......... .......... .......... 49% 41.3M 3s 2024-04-25T03:24:19.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84250K .......... .......... .......... .......... .......... 50% 26.2M 3s 2024-04-25T03:24:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84300K .......... .......... .......... .......... .......... 50% 35.6M 3s 2024-04-25T03:24:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84350K .......... .......... .......... .......... .......... 50% 27.4M 3s 2024-04-25T03:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84400K .......... .......... .......... .......... .......... 50% 21.8M 3s 2024-04-25T03:24:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84450K .......... .......... .......... .......... .......... 50% 40.0M 3s 2024-04-25T03:24:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84500K .......... .......... .......... .......... .......... 50% 25.2M 3s 2024-04-25T03:24:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84550K .......... .......... .......... .......... .......... 50% 23.7M 3s 2024-04-25T03:24:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84600K .......... .......... .......... .......... .......... 50% 43.8M 3s 2024-04-25T03:24:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84650K .......... .......... .......... .......... .......... 50% 24.1M 3s 2024-04-25T03:24:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84700K .......... .......... .......... .......... .......... 50% 40.4M 3s 2024-04-25T03:24:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84750K .......... .......... .......... .......... .......... 50% 24.5M 3s 2024-04-25T03:24:19.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84800K .......... .......... .......... .......... .......... 50% 26.1M 3s 2024-04-25T03:24:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84850K .......... .......... .......... .......... .......... 50% 23.9M 3s 2024-04-25T03:24:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84900K .......... .......... .......... .......... .......... 50% 36.4M 3s 2024-04-25T03:24:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 84950K .......... .......... .......... .......... .......... 50% 44.9M 3s 2024-04-25T03:24:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85000K .......... .......... .......... .......... .......... 50% 23.9M 3s 2024-04-25T03:24:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85050K .......... .......... .......... .......... .......... 50% 41.9M 3s 2024-04-25T03:24:19.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85100K .......... .......... .......... .......... .......... 50% 23.7M 3s 2024-04-25T03:24:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85150K .......... .......... .......... .......... .......... 50% 23.0M 3s 2024-04-25T03:24:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85200K .......... .......... .......... .......... .......... 50% 24.4M 3s 2024-04-25T03:24:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85250K .......... .......... .......... .......... .......... 50% 47.4M 3s 2024-04-25T03:24:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85300K .......... .......... .......... .......... .......... 50% 24.7M 3s 2024-04-25T03:24:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85350K .......... .......... .......... .......... .......... 50% 34.1M 3s 2024-04-25T03:24:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85400K .......... .......... .......... .......... .......... 50% 27.1M 3s 2024-04-25T03:24:19.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85450K .......... .......... .......... .......... .......... 50% 45.6M 3s 2024-04-25T03:24:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85500K .......... .......... .......... .......... .......... 50% 24.6M 3s 2024-04-25T03:24:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85550K .......... .......... .......... .......... .......... 50% 22.6M 3s 2024-04-25T03:24:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85600K .......... .......... .......... .......... .......... 50% 33.5M 3s 2024-04-25T03:24:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85650K .......... .......... .......... .......... .......... 50% 23.5M 3s 2024-04-25T03:24:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85700K .......... .......... .......... .......... .......... 50% 26.2M 3s 2024-04-25T03:24:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85750K .......... .......... .......... .......... .......... 50% 41.1M 3s 2024-04-25T03:24:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85800K .......... .......... .......... .......... .......... 50% 25.5M 3s 2024-04-25T03:24:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85850K .......... .......... .......... .......... .......... 50% 40.8M 3s 2024-04-25T03:24:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85900K .......... .......... .......... .......... .......... 50% 22.2M 3s 2024-04-25T03:24:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 85950K .......... .......... .......... .......... .......... 51% 25.8M 3s 2024-04-25T03:24:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86000K .......... .......... .......... .......... .......... 51% 41.4M 3s 2024-04-25T03:24:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86050K .......... .......... .......... .......... .......... 51% 26.6M 3s 2024-04-25T03:24:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86100K .......... .......... .......... .......... .......... 51% 31.9M 3s 2024-04-25T03:24:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86150K .......... .......... .......... .......... .......... 51% 20.2M 3s 2024-04-25T03:24:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86200K .......... .......... .......... .......... .......... 51% 25.3M 3s 2024-04-25T03:24:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86250K .......... .......... .......... .......... .......... 51% 64.7M 3s 2024-04-25T03:24:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86300K .......... .......... .......... .......... .......... 51% 36.4M 3s 2024-04-25T03:24:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86350K .......... .......... .......... .......... .......... 51% 16.2M 3s 2024-04-25T03:24:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86400K .......... .......... .......... .......... .......... 51% 39.8M 3s 2024-04-25T03:24:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86450K .......... .......... .......... .......... .......... 51% 32.3M 3s 2024-04-25T03:24:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86500K .......... .......... .......... .......... .......... 51% 29.1M 3s 2024-04-25T03:24:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86550K .......... .......... .......... .......... .......... 51% 27.6M 3s 2024-04-25T03:24:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86600K .......... .......... .......... .......... .......... 51% 42.5M 3s 2024-04-25T03:24:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86650K .......... .......... .......... .......... .......... 51% 22.8M 3s 2024-04-25T03:24:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86700K .......... .......... .......... .......... .......... 51% 51.2M 3s 2024-04-25T03:24:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86750K .......... .......... .......... .......... .......... 51% 14.6M 3s 2024-04-25T03:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86800K .......... .......... .......... .......... .......... 51% 67.0M 3s 2024-04-25T03:24:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86850K .......... .......... .......... .......... .......... 51% 30.7M 3s 2024-04-25T03:24:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86900K .......... .......... .......... .......... .......... 51% 25.9M 3s 2024-04-25T03:24:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 86950K .......... .......... .......... .......... .......... 51% 22.1M 3s 2024-04-25T03:24:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87000K .......... .......... .......... .......... .......... 51% 42.9M 3s 2024-04-25T03:24:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87050K .......... .......... .......... .......... .......... 51% 27.1M 3s 2024-04-25T03:24:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87100K .......... .......... .......... .......... .......... 51% 33.9M 3s 2024-04-25T03:24:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87150K .......... .......... .......... .......... .......... 51% 23.5M 3s 2024-04-25T03:24:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87200K .......... .......... .......... .......... .......... 51% 26.3M 3s 2024-04-25T03:24:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87250K .......... .......... .......... .......... .......... 51% 43.5M 3s 2024-04-25T03:24:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87300K .......... .......... .......... .......... .......... 51% 24.7M 3s 2024-04-25T03:24:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87350K .......... .......... .......... .......... .......... 51% 25.6M 3s 2024-04-25T03:24:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87400K .......... .......... .......... .......... .......... 51% 31.6M 3s 2024-04-25T03:24:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87450K .......... .......... .......... .......... .......... 51% 23.0M 3s 2024-04-25T03:24:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87500K .......... .......... .......... .......... .......... 51% 43.7M 3s 2024-04-25T03:24:19.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87550K .......... .......... .......... .......... .......... 51% 27.3M 3s 2024-04-25T03:24:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87600K .......... .......... .......... .......... .......... 51% 22.8M 3s 2024-04-25T03:24:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87650K .......... .......... .......... .......... .......... 52% 28.4M 3s 2024-04-25T03:24:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87700K .......... .......... .......... .......... .......... 52% 28.3M 3s 2024-04-25T03:24:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87750K .......... .......... .......... .......... .......... 52% 41.3M 3s 2024-04-25T03:24:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87800K .......... .......... .......... .......... .......... 52% 31.0M 3s 2024-04-25T03:24:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87850K .......... .......... .......... .......... .......... 52% 27.4M 3s 2024-04-25T03:24:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87900K .......... .......... .......... .......... .......... 52% 25.2M 3s 2024-04-25T03:24:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 87950K .......... .......... .......... .......... .......... 52% 20.6M 3s 2024-04-25T03:24:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88000K .......... .......... .......... .......... .......... 52% 28.2M 3s 2024-04-25T03:24:19.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88050K .......... .......... .......... .......... .......... 52% 52.7M 3s 2024-04-25T03:24:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88100K .......... .......... .......... .......... .......... 52% 22.4M 3s 2024-04-25T03:24:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88150K .......... .......... .......... .......... .......... 52% 31.1M 3s 2024-04-25T03:24:19.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88200K .......... .......... .......... .......... .......... 52% 23.3M 3s 2024-04-25T03:24:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88250K .......... .......... .......... .......... .......... 52% 61.7M 3s 2024-04-25T03:24:19.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88300K .......... .......... .......... .......... .......... 52% 24.1M 3s 2024-04-25T03:24:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88350K .......... .......... .......... .......... .......... 52% 19.8M 3s 2024-04-25T03:24:19.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88400K .......... .......... .......... .......... .......... 52% 61.4M 3s 2024-04-25T03:24:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88450K .......... .......... .......... .......... .......... 52% 22.6M 3s 2024-04-25T03:24:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88500K .......... .......... .......... .......... .......... 52% 25.1M 3s 2024-04-25T03:24:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88550K .......... .......... .......... .......... .......... 52% 51.0M 3s 2024-04-25T03:24:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88600K .......... .......... .......... .......... .......... 52% 21.9M 3s 2024-04-25T03:24:19.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88650K .......... .......... .......... .......... .......... 52% 41.5M 3s 2024-04-25T03:24:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88700K .......... .......... .......... .......... .......... 52% 23.0M 3s 2024-04-25T03:24:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88750K .......... .......... .......... .......... .......... 52% 26.4M 3s 2024-04-25T03:24:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88800K .......... .......... .......... .......... .......... 52% 41.8M 3s 2024-04-25T03:24:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88850K .......... .......... .......... .......... .......... 52% 26.3M 3s 2024-04-25T03:24:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88900K .......... .......... .......... .......... .......... 52% 21.3M 3s 2024-04-25T03:24:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 88950K .......... .......... .......... .......... .......... 52% 41.1M 3s 2024-04-25T03:24:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89000K .......... .......... .......... .......... .......... 52% 55.7M 3s 2024-04-25T03:24:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89050K .......... .......... .......... .......... .......... 52% 22.5M 3s 2024-04-25T03:24:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89100K .......... .......... .......... .......... .......... 52% 34.6M 3s 2024-04-25T03:24:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89150K .......... .......... .......... .......... .......... 52% 17.0M 3s 2024-04-25T03:24:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89200K .......... .......... .......... .......... .......... 52% 28.6M 3s 2024-04-25T03:24:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89250K .......... .......... .......... .......... .......... 52% 19.4M 3s 2024-04-25T03:24:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89300K .......... .......... .......... .......... .......... 53% 61.6M 3s 2024-04-25T03:24:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89350K .......... .......... .......... .......... .......... 53% 21.9M 3s 2024-04-25T03:24:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89400K .......... .......... .......... .......... .......... 53% 21.8M 3s 2024-04-25T03:24:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89450K .......... .......... .......... .......... .......... 53% 16.7M 3s 2024-04-25T03:24:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89500K .......... .......... .......... .......... .......... 53% 42.0M 3s 2024-04-25T03:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89550K .......... .......... .......... .......... .......... 53% 24.3M 3s 2024-04-25T03:24:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89600K .......... .......... .......... .......... .......... 53% 18.6M 3s 2024-04-25T03:24:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89650K .......... .......... .......... .......... .......... 53% 22.9M 3s 2024-04-25T03:24:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89700K .......... .......... .......... .......... .......... 53% 38.0M 3s 2024-04-25T03:24:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89750K .......... .......... .......... .......... .......... 53% 24.2M 3s 2024-04-25T03:24:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89800K .......... .......... .......... .......... .......... 53% 47.5M 3s 2024-04-25T03:24:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89850K .......... .......... .......... .......... .......... 53% 24.5M 3s 2024-04-25T03:24:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89900K .......... .......... .......... .......... .......... 53% 26.1M 3s 2024-04-25T03:24:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 89950K .......... .......... .......... .......... .......... 53% 30.1M 3s 2024-04-25T03:24:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90000K .......... .......... .......... .......... .......... 53% 24.5M 3s 2024-04-25T03:24:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90050K .......... .......... .......... .......... .......... 53% 53.2M 3s 2024-04-25T03:24:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90100K .......... .......... .......... .......... .......... 53% 22.7M 3s 2024-04-25T03:24:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90150K .......... .......... .......... .......... .......... 53% 23.1M 3s 2024-04-25T03:24:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90200K .......... .......... .......... .......... .......... 53% 34.4M 3s 2024-04-25T03:24:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90250K .......... .......... .......... .......... .......... 53% 29.0M 3s 2024-04-25T03:24:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90300K .......... .......... .......... .......... .......... 53% 32.3M 3s 2024-04-25T03:24:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90350K .......... .......... .......... .......... .......... 53% 19.8M 3s 2024-04-25T03:24:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90400K .......... .......... .......... .......... .......... 53% 27.1M 3s 2024-04-25T03:24:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90450K .......... .......... .......... .......... .......... 53% 30.7M 3s 2024-04-25T03:24:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90500K .......... .......... .......... .......... .......... 53% 12.8M 3s 2024-04-25T03:24:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90550K .......... .......... .......... .......... .......... 53% 90.6M 3s 2024-04-25T03:24:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90600K .......... .......... .......... .......... .......... 53% 24.9M 3s 2024-04-25T03:24:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90650K .......... .......... .......... .......... .......... 53% 38.9M 3s 2024-04-25T03:24:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90700K .......... .......... .......... .......... .......... 53% 13.0M 3s 2024-04-25T03:24:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90750K .......... .......... .......... .......... .......... 53% 22.0M 3s 2024-04-25T03:24:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90800K .......... .......... .......... .......... .......... 53% 29.4M 3s 2024-04-25T03:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90850K .......... .......... .......... .......... .......... 53% 40.9M 3s 2024-04-25T03:24:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90900K .......... .......... .......... .......... .......... 53% 24.2M 3s 2024-04-25T03:24:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 90950K .......... .......... .......... .......... .......... 53% 32.3M 3s 2024-04-25T03:24:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91000K .......... .......... .......... .......... .......... 54% 24.9M 3s 2024-04-25T03:24:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91050K .......... .......... .......... .......... .......... 54% 63.9M 3s 2024-04-25T03:24:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91100K .......... .......... .......... .......... .......... 54% 18.4M 3s 2024-04-25T03:24:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91150K .......... .......... .......... .......... .......... 54% 33.2M 3s 2024-04-25T03:24:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91200K .......... .......... .......... .......... .......... 54% 28.6M 3s 2024-04-25T03:24:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91250K .......... .......... .......... .......... .......... 54% 26.9M 3s 2024-04-25T03:24:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91300K .......... .......... .......... .......... .......... 54% 27.1M 3s 2024-04-25T03:24:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91350K .......... .......... .......... .......... .......... 54% 40.4M 3s 2024-04-25T03:24:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91400K .......... .......... .......... .......... .......... 54% 24.3M 3s 2024-04-25T03:24:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91450K .......... .......... .......... .......... .......... 54% 38.7M 3s 2024-04-25T03:24:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91500K .......... .......... .......... .......... .......... 54% 24.5M 3s 2024-04-25T03:24:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91550K .......... .......... .......... .......... .......... 54% 24.6M 3s 2024-04-25T03:24:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91600K .......... .......... .......... .......... .......... 54% 41.4M 3s 2024-04-25T03:24:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91650K .......... .......... .......... .......... .......... 54% 25.0M 3s 2024-04-25T03:24:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91700K .......... .......... .......... .......... .......... 54% 48.1M 3s 2024-04-25T03:24:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91750K .......... .......... .......... .......... .......... 54% 23.2M 3s 2024-04-25T03:24:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91800K .......... .......... .......... .......... .......... 54% 44.1M 3s 2024-04-25T03:24:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91850K .......... .......... .......... .......... .......... 54% 19.9M 3s 2024-04-25T03:24:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91900K .......... .......... .......... .......... .......... 54% 29.9M 3s 2024-04-25T03:24:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 91950K .......... .......... .......... .......... .......... 54% 22.1M 3s 2024-04-25T03:24:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92000K .......... .......... .......... .......... .......... 54% 39.7M 3s 2024-04-25T03:24:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92050K .......... .......... .......... .......... .......... 54% 41.6M 3s 2024-04-25T03:24:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92100K .......... .......... .......... .......... .......... 54% 19.9M 3s 2024-04-25T03:24:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92150K .......... .......... .......... .......... .......... 54% 40.5M 3s 2024-04-25T03:24:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92200K .......... .......... .......... .......... .......... 54% 33.1M 3s 2024-04-25T03:24:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92250K .......... .......... .......... .......... .......... 54% 31.6M 3s 2024-04-25T03:24:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92300K .......... .......... .......... .......... .......... 54% 36.8M 3s 2024-04-25T03:24:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92350K .......... .......... .......... .......... .......... 54% 19.6M 3s 2024-04-25T03:24:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92400K .......... .......... .......... .......... .......... 54% 27.0M 3s 2024-04-25T03:24:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92450K .......... .......... .......... .......... .......... 54% 21.3M 3s 2024-04-25T03:24:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92500K .......... .......... .......... .......... .......... 54% 27.8M 3s 2024-04-25T03:24:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92550K .......... .......... .......... .......... .......... 54% 22.8M 3s 2024-04-25T03:24:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92600K .......... .......... .......... .......... .......... 54% 38.3M 3s 2024-04-25T03:24:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92650K .......... .......... .......... .......... .......... 54% 27.9M 3s 2024-04-25T03:24:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92700K .......... .......... .......... .......... .......... 55% 41.7M 3s 2024-04-25T03:24:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92750K .......... .......... .......... .......... .......... 55% 8.99M 3s 2024-04-25T03:24:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92800K .......... .......... .......... .......... .......... 55% 24.9M 3s 2024-04-25T03:24:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92850K .......... .......... .......... .......... .......... 55% 43.5M 3s 2024-04-25T03:24:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92900K .......... .......... .......... .......... .......... 55% 15.7M 3s 2024-04-25T03:24:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 92950K .......... .......... .......... .......... .......... 55% 36.3M 3s 2024-04-25T03:24:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93000K .......... .......... .......... .......... .......... 55% 25.1M 3s 2024-04-25T03:24:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93050K .......... .......... .......... .......... .......... 55% 22.9M 3s 2024-04-25T03:24:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93100K .......... .......... .......... .......... .......... 55% 42.9M 3s 2024-04-25T03:24:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93150K .......... .......... .......... .......... .......... 55% 23.6M 3s 2024-04-25T03:24:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93200K .......... .......... .......... .......... .......... 55% 22.5M 3s 2024-04-25T03:24:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93250K .......... .......... .......... .......... .......... 55% 38.9M 3s 2024-04-25T03:24:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93300K .......... .......... .......... .......... .......... 55% 27.2M 3s 2024-04-25T03:24:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93350K .......... .......... .......... .......... .......... 55% 43.1M 3s 2024-04-25T03:24:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93400K .......... .......... .......... .......... .......... 55% 23.0M 3s 2024-04-25T03:24:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93450K .......... .......... .......... .......... .......... 55% 40.2M 3s 2024-04-25T03:24:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93500K .......... .......... .......... .......... .......... 55% 23.2M 3s 2024-04-25T03:24:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93550K .......... .......... .......... .......... .......... 55% 31.8M 3s 2024-04-25T03:24:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93600K .......... .......... .......... .......... .......... 55% 23.2M 3s 2024-04-25T03:24:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93650K .......... .......... .......... .......... .......... 55% 34.0M 3s 2024-04-25T03:24:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93700K .......... .......... .......... .......... .......... 55% 24.0M 3s 2024-04-25T03:24:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93750K .......... .......... .......... .......... .......... 55% 36.5M 3s 2024-04-25T03:24:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93800K .......... .......... .......... .......... .......... 55% 24.0M 3s 2024-04-25T03:24:19.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93850K .......... .......... .......... .......... .......... 55% 40.9M 3s 2024-04-25T03:24:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93900K .......... .......... .......... .......... .......... 55% 27.9M 3s 2024-04-25T03:24:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 93950K .......... .......... .......... .......... .......... 55% 22.5M 3s 2024-04-25T03:24:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94000K .......... .......... .......... .......... .......... 55% 34.0M 3s 2024-04-25T03:24:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94050K .......... .......... .......... .......... .......... 55% 27.1M 3s 2024-04-25T03:24:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94100K .......... .......... .......... .......... .......... 55% 40.5M 3s 2024-04-25T03:24:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94150K .......... .......... .......... .......... .......... 55% 24.1M 3s 2024-04-25T03:24:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94200K .......... .......... .......... .......... .......... 55% 23.9M 3s 2024-04-25T03:24:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94250K .......... .......... .......... .......... .......... 55% 41.6M 3s 2024-04-25T03:24:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94300K .......... .......... .......... .......... .......... 55% 25.9M 3s 2024-04-25T03:24:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94350K .......... .......... .......... .......... .......... 56% 21.8M 3s 2024-04-25T03:24:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94400K .......... .......... .......... .......... .......... 56% 40.6M 3s 2024-04-25T03:24:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94450K .......... .......... .......... .......... .......... 56% 28.0M 3s 2024-04-25T03:24:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94500K .......... .......... .......... .......... .......... 56% 40.7M 3s 2024-04-25T03:24:19.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94550K .......... .......... .......... .......... .......... 56% 23.9M 3s 2024-04-25T03:24:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94600K .......... .......... .......... .......... .......... 56% 47.7M 3s 2024-04-25T03:24:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94650K .......... .......... .......... .......... .......... 56% 22.4M 3s 2024-04-25T03:24:19.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94700K .......... .......... .......... .......... .......... 56% 43.9M 3s 2024-04-25T03:24:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94750K .......... .......... .......... .......... .......... 56% 15.4M 3s 2024-04-25T03:24:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94800K .......... .......... .......... .......... .......... 56% 34.4M 3s 2024-04-25T03:24:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94850K .......... .......... .......... .......... .......... 56% 27.4M 3s 2024-04-25T03:24:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94900K .......... .......... .......... .......... .......... 56% 43.3M 3s 2024-04-25T03:24:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 94950K .......... .......... .......... .......... .......... 56% 24.1M 3s 2024-04-25T03:24:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95000K .......... .......... .......... .......... .......... 56% 43.8M 3s 2024-04-25T03:24:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95050K .......... .......... .......... .......... .......... 56% 24.4M 3s 2024-04-25T03:24:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95100K .......... .......... .......... .......... .......... 56% 38.8M 3s 2024-04-25T03:24:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95150K .......... .......... .......... .......... .......... 56% 22.7M 3s 2024-04-25T03:24:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95200K .......... .......... .......... .......... .......... 56% 24.3M 3s 2024-04-25T03:24:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95250K .......... .......... .......... .......... .......... 56% 23.5M 3s 2024-04-25T03:24:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95300K .......... .......... .......... .......... .......... 56% 42.6M 3s 2024-04-25T03:24:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95350K .......... .......... .......... .......... .......... 56% 25.1M 3s 2024-04-25T03:24:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95400K .......... .......... .......... .......... .......... 56% 38.8M 3s 2024-04-25T03:24:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95450K .......... .......... .......... .......... .......... 56% 24.6M 3s 2024-04-25T03:24:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95500K .......... .......... .......... .......... .......... 56% 40.1M 3s 2024-04-25T03:24:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95550K .......... .......... .......... .......... .......... 56% 18.9M 3s 2024-04-25T03:24:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95600K .......... .......... .......... .......... .......... 56% 28.7M 3s 2024-04-25T03:24:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95650K .......... .......... .......... .......... .......... 56% 38.6M 3s 2024-04-25T03:24:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95700K .......... .......... .......... .......... .......... 56% 26.7M 3s 2024-04-25T03:24:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95750K .......... .......... .......... .......... .......... 56% 40.7M 3s 2024-04-25T03:24:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95800K .......... .......... .......... .......... .......... 56% 24.9M 3s 2024-04-25T03:24:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95850K .......... .......... .......... .......... .......... 56% 21.8M 3s 2024-04-25T03:24:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95900K .......... .......... .......... .......... .......... 56% 39.3M 3s 2024-04-25T03:24:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 95950K .......... .......... .......... .......... .......... 56% 26.2M 3s 2024-04-25T03:24:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96000K .......... .......... .......... .......... .......... 56% 23.4M 3s 2024-04-25T03:24:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96050K .......... .......... .......... .......... .......... 57% 43.7M 3s 2024-04-25T03:24:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96100K .......... .......... .......... .......... .......... 57% 23.1M 3s 2024-04-25T03:24:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96150K .......... .......... .......... .......... .......... 57% 39.3M 3s 2024-04-25T03:24:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96200K .......... .......... .......... .......... .......... 57% 23.4M 3s 2024-04-25T03:24:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96250K .......... .......... .......... .......... .......... 57% 47.4M 3s 2024-04-25T03:24:19.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96300K .......... .......... .......... .......... .......... 57% 24.0M 3s 2024-04-25T03:24:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96350K .......... .......... .......... .......... .......... 57% 24.4M 3s 2024-04-25T03:24:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96400K .......... .......... .......... .......... .......... 57% 22.0M 3s 2024-04-25T03:24:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96450K .......... .......... .......... .......... .......... 57% 49.4M 3s 2024-04-25T03:24:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96500K .......... .......... .......... .......... .......... 57% 23.8M 3s 2024-04-25T03:24:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96550K .......... .......... .......... .......... .......... 57% 42.5M 3s 2024-04-25T03:24:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96600K .......... .......... .......... .......... .......... 57% 22.6M 3s 2024-04-25T03:24:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96650K .......... .......... .......... .......... .......... 57% 51.4M 3s 2024-04-25T03:24:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96700K .......... .......... .......... .......... .......... 57% 26.1M 3s 2024-04-25T03:24:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96750K .......... .......... .......... .......... .......... 57% 22.6M 3s 2024-04-25T03:24:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96800K .......... .......... .......... .......... .......... 57% 20.7M 3s 2024-04-25T03:24:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96850K .......... .......... .......... .......... .......... 57% 108M 3s 2024-04-25T03:24:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96900K .......... .......... .......... .......... .......... 57% 29.8M 3s 2024-04-25T03:24:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 96950K .......... .......... .......... .......... .......... 57% 21.7M 3s 2024-04-25T03:24:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97000K .......... .......... .......... .......... .......... 57% 37.8M 3s 2024-04-25T03:24:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97050K .......... .......... .......... .......... .......... 57% 21.8M 3s 2024-04-25T03:24:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97100K .......... .......... .......... .......... .......... 57% 26.0M 3s 2024-04-25T03:24:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97150K .......... .......... .......... .......... .......... 57% 30.2M 3s 2024-04-25T03:24:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97200K .......... .......... .......... .......... .......... 57% 38.2M 3s 2024-04-25T03:24:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97250K .......... .......... .......... .......... .......... 57% 23.7M 3s 2024-04-25T03:24:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97300K .......... .......... .......... .......... .......... 57% 31.3M 3s 2024-04-25T03:24:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97350K .......... .......... .......... .......... .......... 57% 27.1M 3s 2024-04-25T03:24:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97400K .......... .......... .......... .......... .......... 57% 39.1M 3s 2024-04-25T03:24:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97450K .......... .......... .......... .......... .......... 57% 28.4M 3s 2024-04-25T03:24:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97500K .......... .......... .......... .......... .......... 57% 23.4M 3s 2024-04-25T03:24:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97550K .......... .......... .......... .......... .......... 57% 37.2M 3s 2024-04-25T03:24:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97600K .......... .......... .......... .......... .......... 57% 23.5M 3s 2024-04-25T03:24:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97650K .......... .......... .......... .......... .......... 57% 26.2M 3s 2024-04-25T03:24:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97700K .......... .......... .......... .......... .......... 57% 37.4M 3s 2024-04-25T03:24:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97750K .......... .......... .......... .......... .......... 58% 27.4M 3s 2024-04-25T03:24:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97800K .......... .......... .......... .......... .......... 58% 39.3M 3s 2024-04-25T03:24:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97850K .......... .......... .......... .......... .......... 58% 35.6M 3s 2024-04-25T03:24:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97900K .......... .......... .......... .......... .......... 58% 31.2M 3s 2024-04-25T03:24:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 97950K .......... .......... .......... .......... .......... 58% 22.9M 3s 2024-04-25T03:24:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98000K .......... .......... .......... .......... .......... 58% 22.4M 3s 2024-04-25T03:24:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98050K .......... .......... .......... .......... .......... 58% 39.4M 3s 2024-04-25T03:24:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98100K .......... .......... .......... .......... .......... 58% 20.3M 3s 2024-04-25T03:24:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98150K .......... .......... .......... .......... .......... 58% 25.5M 3s 2024-04-25T03:24:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98200K .......... .......... .......... .......... .......... 58% 61.2M 3s 2024-04-25T03:24:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98250K .......... .......... .......... .......... .......... 58% 22.8M 3s 2024-04-25T03:24:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98300K .......... .......... .......... .......... .......... 58% 32.0M 3s 2024-04-25T03:24:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98350K .......... .......... .......... .......... .......... 58% 23.6M 3s 2024-04-25T03:24:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98400K .......... .......... .......... .......... .......... 58% 27.6M 3s 2024-04-25T03:24:20.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98450K .......... .......... .......... .......... .......... 58% 50.3M 3s 2024-04-25T03:24:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98500K .......... .......... .......... .......... .......... 58% 24.7M 3s 2024-04-25T03:24:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98550K .......... .......... .......... .......... .......... 58% 36.3M 3s 2024-04-25T03:24:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98600K .......... .......... .......... .......... .......... 58% 26.1M 3s 2024-04-25T03:24:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98650K .......... .......... .......... .......... .......... 58% 38.0M 3s 2024-04-25T03:24:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98700K .......... .......... .......... .......... .......... 58% 26.9M 3s 2024-04-25T03:24:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98750K .......... .......... .......... .......... .......... 58% 29.8M 3s 2024-04-25T03:24:20.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98800K .......... .......... .......... .......... .......... 58% 18.8M 3s 2024-04-25T03:24:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98850K .......... .......... .......... .......... .......... 58% 37.1M 3s 2024-04-25T03:24:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98900K .......... .......... .......... .......... .......... 58% 26.1M 3s 2024-04-25T03:24:20.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 98950K .......... .......... .......... .......... .......... 58% 41.1M 3s 2024-04-25T03:24:20.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99000K .......... .......... .......... .......... .......... 58% 26.7M 3s 2024-04-25T03:24:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99050K .......... .......... .......... .......... .......... 58% 34.1M 3s 2024-04-25T03:24:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99100K .......... .......... .......... .......... .......... 58% 26.2M 3s 2024-04-25T03:24:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99150K .......... .......... .......... .......... .......... 58% 22.3M 3s 2024-04-25T03:24:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99200K .......... .......... .......... .......... .......... 58% 37.1M 3s 2024-04-25T03:24:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99250K .......... .......... .......... .......... .......... 58% 35.5M 3s 2024-04-25T03:24:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99300K .......... .......... .......... .......... .......... 58% 26.9M 3s 2024-04-25T03:24:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99350K .......... .......... .......... .......... .......... 58% 26.2M 3s 2024-04-25T03:24:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99400K .......... .......... .......... .......... .......... 58% 19.0M 3s 2024-04-25T03:24:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99450K .......... .......... .......... .......... .......... 59% 113M 3s 2024-04-25T03:24:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99500K .......... .......... .......... .......... .......... 59% 23.9M 3s 2024-04-25T03:24:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99550K .......... .......... .......... .......... .......... 59% 24.1M 3s 2024-04-25T03:24:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99600K .......... .......... .......... .......... .......... 59% 37.8M 3s 2024-04-25T03:24:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99650K .......... .......... .......... .......... .......... 59% 25.2M 3s 2024-04-25T03:24:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99700K .......... .......... .......... .......... .......... 59% 46.1M 3s 2024-04-25T03:24:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99750K .......... .......... .......... .......... .......... 59% 26.8M 3s 2024-04-25T03:24:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99800K .......... .......... .......... .......... .......... 59% 24.3M 3s 2024-04-25T03:24:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99850K .......... .......... .......... .......... .......... 59% 28.0M 3s 2024-04-25T03:24:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99900K .......... .......... .......... .......... .......... 59% 22.8M 2s 2024-04-25T03:24:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 99950K .......... .......... .......... .......... .......... 59% 25.9M 2s 2024-04-25T03:24:20.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100000K .......... .......... .......... .......... .......... 59% 43.7M 2s 2024-04-25T03:24:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100050K .......... .......... .......... .......... .......... 59% 25.2M 2s 2024-04-25T03:24:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100100K .......... .......... .......... .......... .......... 59% 34.1M 2s 2024-04-25T03:24:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100150K .......... .......... .......... .......... .......... 59% 27.0M 2s 2024-04-25T03:24:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100200K .......... .......... .......... .......... .......... 59% 36.9M 2s 2024-04-25T03:24:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100250K .......... .......... .......... .......... .......... 59% 27.6M 2s 2024-04-25T03:24:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100300K .......... .......... .......... .......... .......... 59% 32.4M 2s 2024-04-25T03:24:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100350K .......... .......... .......... .......... .......... 59% 28.2M 2s 2024-04-25T03:24:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100400K .......... .......... .......... .......... .......... 59% 23.5M 2s 2024-04-25T03:24:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100450K .......... .......... .......... .......... .......... 59% 23.9M 2s 2024-04-25T03:24:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100500K .......... .......... .......... .......... .......... 59% 46.2M 2s 2024-04-25T03:24:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100550K .......... .......... .......... .......... .......... 59% 24.7M 2s 2024-04-25T03:24:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100600K .......... .......... .......... .......... .......... 59% 39.4M 2s 2024-04-25T03:24:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100650K .......... .......... .......... .......... .......... 59% 22.7M 2s 2024-04-25T03:24:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100700K .......... .......... .......... .......... .......... 59% 56.2M 2s 2024-04-25T03:24:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100750K .......... .......... .......... .......... .......... 59% 23.9M 2s 2024-04-25T03:24:20.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100800K .......... .......... .......... .......... .......... 59% 21.4M 2s 2024-04-25T03:24:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100850K .......... .......... .......... .......... .......... 59% 41.9M 2s 2024-04-25T03:24:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100900K .......... .......... .......... .......... .......... 59% 23.0M 2s 2024-04-25T03:24:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:100950K .......... .......... .......... .......... .......... 59% 22.0M 2s 2024-04-25T03:24:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101000K .......... .......... .......... .......... .......... 59% 55.6M 2s 2024-04-25T03:24:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101050K .......... .......... .......... .......... .......... 59% 23.8M 2s 2024-04-25T03:24:20.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101100K .......... .......... .......... .......... .......... 60% 32.2M 2s 2024-04-25T03:24:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101150K .......... .......... .......... .......... .......... 60% 27.8M 2s 2024-04-25T03:24:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101200K .......... .......... .......... .......... .......... 60% 21.2M 2s 2024-04-25T03:24:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101250K .......... .......... .......... .......... .......... 60% 44.1M 2s 2024-04-25T03:24:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101300K .......... .......... .......... .......... .......... 60% 23.6M 2s 2024-04-25T03:24:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101350K .......... .......... .......... .......... .......... 60% 42.0M 2s 2024-04-25T03:24:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101400K .......... .......... .......... .......... .......... 60% 28.1M 2s 2024-04-25T03:24:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101450K .......... .......... .......... .......... .......... 60% 34.9M 2s 2024-04-25T03:24:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101500K .......... .......... .......... .......... .......... 60% 25.6M 2s 2024-04-25T03:24:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101550K .......... .......... .......... .......... .......... 60% 24.5M 2s 2024-04-25T03:24:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101600K .......... .......... .......... .......... .......... 60% 22.9M 2s 2024-04-25T03:24:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101650K .......... .......... .......... .......... .......... 60% 35.5M 2s 2024-04-25T03:24:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101700K .......... .......... .......... .......... .......... 60% 25.2M 2s 2024-04-25T03:24:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101750K .......... .......... .......... .......... .......... 60% 48.3M 2s 2024-04-25T03:24:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101800K .......... .......... .......... .......... .......... 60% 23.3M 2s 2024-04-25T03:24:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101850K .......... .......... .......... .......... .......... 60% 39.2M 2s 2024-04-25T03:24:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101900K .......... .......... .......... .......... .......... 60% 31.2M 2s 2024-04-25T03:24:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:101950K .......... .......... .......... .......... .......... 60% 17.3M 2s 2024-04-25T03:24:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102000K .......... .......... .......... .......... .......... 60% 110M 2s 2024-04-25T03:24:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102050K .......... .......... .......... .......... .......... 60% 22.7M 2s 2024-04-25T03:24:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102100K .......... .......... .......... .......... .......... 60% 24.1M 2s 2024-04-25T03:24:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102150K .......... .......... .......... .......... .......... 60% 30.9M 2s 2024-04-25T03:24:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102200K .......... .......... .......... .......... .......... 60% 51.5M 2s 2024-04-25T03:24:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102250K .......... .......... .......... .......... .......... 60% 25.9M 2s 2024-04-25T03:24:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102300K .......... .......... .......... .......... .......... 60% 24.0M 2s 2024-04-25T03:24:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102350K .......... .......... .......... .......... .......... 60% 23.6M 2s 2024-04-25T03:24:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102400K .......... .......... .......... .......... .......... 60% 40.7M 2s 2024-04-25T03:24:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102450K .......... .......... .......... .......... .......... 60% 22.2M 2s 2024-04-25T03:24:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102500K .......... .......... .......... .......... .......... 60% 62.0M 2s 2024-04-25T03:24:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102550K .......... .......... .......... .......... .......... 60% 20.7M 2s 2024-04-25T03:24:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102600K .......... .......... .......... .......... .......... 60% 43.9M 2s 2024-04-25T03:24:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102650K .......... .......... .......... .......... .......... 60% 18.0M 2s 2024-04-25T03:24:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102700K .......... .......... .......... .......... .......... 60% 36.7M 2s 2024-04-25T03:24:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102750K .......... .......... .......... .......... .......... 60% 23.2M 2s 2024-04-25T03:24:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102800K .......... .......... .......... .......... .......... 61% 49.1M 2s 2024-04-25T03:24:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102850K .......... .......... .......... .......... .......... 61% 23.7M 2s 2024-04-25T03:24:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102900K .......... .......... .......... .......... .......... 61% 33.5M 2s 2024-04-25T03:24:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:102950K .......... .......... .......... .......... .......... 61% 27.8M 2s 2024-04-25T03:24:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103000K .......... .......... .......... .......... .......... 61% 44.3M 2s 2024-04-25T03:24:20.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103050K .......... .......... .......... .......... .......... 61% 25.7M 2s 2024-04-25T03:24:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103100K .......... .......... .......... .......... .......... 61% 40.2M 2s 2024-04-25T03:24:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103150K .......... .......... .......... .......... .......... 61% 18.9M 2s 2024-04-25T03:24:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103200K .......... .......... .......... .......... .......... 61% 25.0M 2s 2024-04-25T03:24:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103250K .......... .......... .......... .......... .......... 61% 28.1M 2s 2024-04-25T03:24:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103300K .......... .......... .......... .......... .......... 61% 32.5M 2s 2024-04-25T03:24:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103350K .......... .......... .......... .......... .......... 61% 20.6M 2s 2024-04-25T03:24:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103400K .......... .......... .......... .......... .......... 61% 28.9M 2s 2024-04-25T03:24:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103450K .......... .......... .......... .......... .......... 61% 24.2M 2s 2024-04-25T03:24:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103500K .......... .......... .......... .......... .......... 61% 39.5M 2s 2024-04-25T03:24:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103550K .......... .......... .......... .......... .......... 61% 7.92M 2s 2024-04-25T03:24:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103600K .......... .......... .......... .......... .......... 61% 21.4M 2s 2024-04-25T03:24:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103650K .......... .......... .......... .......... .......... 61% 31.4M 2s 2024-04-25T03:24:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103700K .......... .......... .......... .......... .......... 61% 27.8M 2s 2024-04-25T03:24:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103750K .......... .......... .......... .......... .......... 61% 20.2M 2s 2024-04-25T03:24:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103800K .......... .......... .......... .......... .......... 61% 72.3M 2s 2024-04-25T03:24:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103850K .......... .......... .......... .......... .......... 61% 21.5M 2s 2024-04-25T03:24:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103900K .......... .......... .......... .......... .......... 61% 40.9M 2s 2024-04-25T03:24:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:103950K .......... .......... .......... .......... .......... 61% 20.3M 2s 2024-04-25T03:24:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104000K .......... .......... .......... .......... .......... 61% 26.0M 2s 2024-04-25T03:24:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104050K .......... .......... .......... .......... .......... 61% 48.1M 2s 2024-04-25T03:24:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104100K .......... .......... .......... .......... .......... 61% 25.7M 2s 2024-04-25T03:24:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104150K .......... .......... .......... .......... .......... 61% 23.0M 2s 2024-04-25T03:24:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104200K .......... .......... .......... .......... .......... 61% 41.8M 2s 2024-04-25T03:24:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104250K .......... .......... .......... .......... .......... 61% 48.9M 2s 2024-04-25T03:24:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104300K .......... .......... .......... .......... .......... 61% 22.3M 2s 2024-04-25T03:24:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104350K .......... .......... .......... .......... .......... 61% 23.0M 2s 2024-04-25T03:24:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104400K .......... .......... .......... .......... .......... 61% 40.0M 2s 2024-04-25T03:24:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104450K .......... .......... .......... .......... .......... 61% 22.7M 2s 2024-04-25T03:24:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104500K .......... .......... .......... .......... .......... 62% 25.3M 2s 2024-04-25T03:24:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104550K .......... .......... .......... .......... .......... 62% 37.0M 2s 2024-04-25T03:24:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104600K .......... .......... .......... .......... .......... 62% 24.1M 2s 2024-04-25T03:24:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104650K .......... .......... .......... .......... .......... 62% 44.9M 2s 2024-04-25T03:24:20.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104700K .......... .......... .......... .......... .......... 62% 22.6M 2s 2024-04-25T03:24:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104750K .......... .......... .......... .......... .......... 62% 24.9M 2s 2024-04-25T03:24:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104800K .......... .......... .......... .......... .......... 62% 50.2M 2s 2024-04-25T03:24:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104850K .......... .......... .......... .......... .......... 62% 23.8M 2s 2024-04-25T03:24:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104900K .......... .......... .......... .......... .......... 62% 42.4M 2s 2024-04-25T03:24:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:104950K .......... .......... .......... .......... .......... 62% 20.5M 2s 2024-04-25T03:24:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105000K .......... .......... .......... .......... .......... 62% 26.1M 2s 2024-04-25T03:24:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105050K .......... .......... .......... .......... .......... 62% 48.8M 2s 2024-04-25T03:24:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105100K .......... .......... .......... .......... .......... 62% 25.2M 2s 2024-04-25T03:24:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105150K .......... .......... .......... .......... .......... 62% 21.2M 2s 2024-04-25T03:24:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105200K .......... .......... .......... .......... .......... 62% 41.7M 2s 2024-04-25T03:24:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105250K .......... .......... .......... .......... .......... 62% 24.0M 2s 2024-04-25T03:24:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105300K .......... .......... .......... .......... .......... 62% 35.9M 2s 2024-04-25T03:24:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105350K .......... .......... .......... .......... .......... 62% 16.5M 2s 2024-04-25T03:24:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105400K .......... .......... .......... .......... .......... 62% 32.7M 2s 2024-04-25T03:24:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105450K .......... .......... .......... .......... .......... 62% 13.9M 2s 2024-04-25T03:24:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105500K .......... .......... .......... .......... .......... 62% 24.6M 2s 2024-04-25T03:24:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105550K .......... .......... .......... .......... .......... 62% 20.7M 2s 2024-04-25T03:24:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105600K .......... .......... .......... .......... .......... 62% 47.9M 2s 2024-04-25T03:24:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105650K .......... .......... .......... .......... .......... 62% 24.6M 2s 2024-04-25T03:24:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105700K .......... .......... .......... .......... .......... 62% 35.7M 2s 2024-04-25T03:24:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105750K .......... .......... .......... .......... .......... 62% 24.8M 2s 2024-04-25T03:24:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105800K .......... .......... .......... .......... .......... 62% 14.2M 2s 2024-04-25T03:24:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105850K .......... .......... .......... .......... .......... 62% 19.1M 2s 2024-04-25T03:24:20.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105900K .......... .......... .......... .......... .......... 62% 52.0M 2s 2024-04-25T03:24:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:105950K .......... .......... .......... .......... .......... 62% 22.8M 2s 2024-04-25T03:24:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106000K .......... .......... .......... .......... .......... 62% 18.0M 2s 2024-04-25T03:24:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106050K .......... .......... .......... .......... .......... 62% 23.6M 2s 2024-04-25T03:24:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106100K .......... .......... .......... .......... .......... 62% 39.7M 2s 2024-04-25T03:24:20.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106150K .......... .......... .......... .......... .......... 63% 24.3M 2s 2024-04-25T03:24:20.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106200K .......... .......... .......... .......... .......... 63% 27.3M 2s 2024-04-25T03:24:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106250K .......... .......... .......... .......... .......... 63% 22.8M 2s 2024-04-25T03:24:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106300K .......... .......... .......... .......... .......... 63% 37.1M 2s 2024-04-25T03:24:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106350K .......... .......... .......... .......... .......... 63% 23.1M 2s 2024-04-25T03:24:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106400K .......... .......... .......... .......... .......... 63% 27.2M 2s 2024-04-25T03:24:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106450K .......... .......... .......... .......... .......... 63% 44.5M 2s 2024-04-25T03:24:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106500K .......... .......... .......... .......... .......... 63% 21.7M 2s 2024-04-25T03:24:20.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106550K .......... .......... .......... .......... .......... 63% 39.6M 2s 2024-04-25T03:24:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106600K .......... .......... .......... .......... .......... 63% 37.1M 2s 2024-04-25T03:24:20.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106650K .......... .......... .......... .......... .......... 63% 18.8M 2s 2024-04-25T03:24:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106700K .......... .......... .......... .......... .......... 63% 43.9M 2s 2024-04-25T03:24:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106750K .......... .......... .......... .......... .......... 63% 23.2M 2s 2024-04-25T03:24:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106800K .......... .......... .......... .......... .......... 63% 26.2M 2s 2024-04-25T03:24:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106850K .......... .......... .......... .......... .......... 63% 38.8M 2s 2024-04-25T03:24:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106900K .......... .......... .......... .......... .......... 63% 22.6M 2s 2024-04-25T03:24:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:106950K .......... .......... .......... .......... .......... 63% 34.6M 2s 2024-04-25T03:24:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107000K .......... .......... .......... .......... .......... 63% 28.8M 2s 2024-04-25T03:24:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107050K .......... .......... .......... .......... .......... 63% 49.8M 2s 2024-04-25T03:24:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107100K .......... .......... .......... .......... .......... 63% 22.0M 2s 2024-04-25T03:24:20.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107150K .......... .......... .......... .......... .......... 63% 31.4M 2s 2024-04-25T03:24:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107200K .......... .......... .......... .......... .......... 63% 18.4M 2s 2024-04-25T03:24:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107250K .......... .......... .......... .......... .......... 63% 46.3M 2s 2024-04-25T03:24:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107300K .......... .......... .......... .......... .......... 63% 22.2M 2s 2024-04-25T03:24:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107350K .......... .......... .......... .......... .......... 63% 43.7M 2s 2024-04-25T03:24:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107400K .......... .......... .......... .......... .......... 63% 24.5M 2s 2024-04-25T03:24:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107450K .......... .......... .......... .......... .......... 63% 38.1M 2s 2024-04-25T03:24:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107500K .......... .......... .......... .......... .......... 63% 21.8M 2s 2024-04-25T03:24:20.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107550K .......... .......... .......... .......... .......... 63% 26.8M 2s 2024-04-25T03:24:20.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107600K .......... .......... .......... .......... .......... 63% 34.6M 2s 2024-04-25T03:24:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107650K .......... .......... .......... .......... .......... 63% 29.3M 2s 2024-04-25T03:24:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107700K .......... .......... .......... .......... .......... 63% 40.8M 2s 2024-04-25T03:24:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107750K .......... .......... .......... .......... .......... 63% 22.8M 2s 2024-04-25T03:24:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107800K .......... .......... .......... .......... .......... 63% 24.3M 2s 2024-04-25T03:24:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107850K .......... .......... .......... .......... .......... 64% 36.2M 2s 2024-04-25T03:24:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107900K .......... .......... .......... .......... .......... 64% 27.6M 2s 2024-04-25T03:24:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:107950K .......... .......... .......... .......... .......... 64% 26.3M 2s 2024-04-25T03:24:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108000K .......... .......... .......... .......... .......... 64% 33.4M 2s 2024-04-25T03:24:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108050K .......... .......... .......... .......... .......... 64% 24.1M 2s 2024-04-25T03:24:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108100K .......... .......... .......... .......... .......... 64% 41.1M 2s 2024-04-25T03:24:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108150K .......... .......... .......... .......... .......... 64% 25.6M 2s 2024-04-25T03:24:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108200K .......... .......... .......... .......... .......... 64% 40.9M 2s 2024-04-25T03:24:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108250K .......... .......... .......... .......... .......... 64% 24.4M 2s 2024-04-25T03:24:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108300K .......... .......... .......... .......... .......... 64% 22.1M 2s 2024-04-25T03:24:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108350K .......... .......... .......... .......... .......... 64% 25.9M 2s 2024-04-25T03:24:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108400K .......... .......... .......... .......... .......... 64% 43.2M 2s 2024-04-25T03:24:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108450K .......... .......... .......... .......... .......... 64% 23.4M 2s 2024-04-25T03:24:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108500K .......... .......... .......... .......... .......... 64% 44.7M 2s 2024-04-25T03:24:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108550K .......... .......... .......... .......... .......... 64% 13.4M 2s 2024-04-25T03:24:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108600K .......... .......... .......... .......... .......... 64% 148M 2s 2024-04-25T03:24:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108650K .......... .......... .......... .......... .......... 64% 41.5M 2s 2024-04-25T03:24:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108700K .......... .......... .......... .......... .......... 64% 32.1M 2s 2024-04-25T03:24:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108750K .......... .......... .......... .......... .......... 64% 28.7M 2s 2024-04-25T03:24:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108800K .......... .......... .......... .......... .......... 64% 25.2M 2s 2024-04-25T03:24:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108850K .......... .......... .......... .......... .......... 64% 20.9M 2s 2024-04-25T03:24:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108900K .......... .......... .......... .......... .......... 64% 42.4M 2s 2024-04-25T03:24:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:108950K .......... .......... .......... .......... .......... 64% 23.3M 2s 2024-04-25T03:24:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109000K .......... .......... .......... .......... .......... 64% 48.2M 2s 2024-04-25T03:24:20.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109050K .......... .......... .......... .......... .......... 64% 24.7M 2s 2024-04-25T03:24:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109100K .......... .......... .......... .......... .......... 64% 42.5M 2s 2024-04-25T03:24:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109150K .......... .......... .......... .......... .......... 64% 21.5M 2s 2024-04-25T03:24:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109200K .......... .......... .......... .......... .......... 64% 26.1M 2s 2024-04-25T03:24:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109250K .......... .......... .......... .......... .......... 64% 45.6M 2s 2024-04-25T03:24:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109300K .......... .......... .......... .......... .......... 64% 24.4M 2s 2024-04-25T03:24:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109350K .......... .......... .......... .......... .......... 64% 47.2M 2s 2024-04-25T03:24:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109400K .......... .......... .......... .......... .......... 64% 30.4M 2s 2024-04-25T03:24:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109450K .......... .......... .......... .......... .......... 64% 18.4M 2s 2024-04-25T03:24:20.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109500K .......... .......... .......... .......... .......... 64% 37.3M 2s 2024-04-25T03:24:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109550K .......... .......... .......... .......... .......... 65% 21.7M 2s 2024-04-25T03:24:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109600K .......... .......... .......... .......... .......... 65% 28.5M 2s 2024-04-25T03:24:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109650K .......... .......... .......... .......... .......... 65% 46.1M 2s 2024-04-25T03:24:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109700K .......... .......... .......... .......... .......... 65% 24.2M 2s 2024-04-25T03:24:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109750K .......... .......... .......... .......... .......... 65% 39.6M 2s 2024-04-25T03:24:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109800K .......... .......... .......... .......... .......... 65% 23.6M 2s 2024-04-25T03:24:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109850K .......... .......... .......... .......... .......... 65% 39.9M 2s 2024-04-25T03:24:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109900K .......... .......... .......... .......... .......... 65% 34.4M 2s 2024-04-25T03:24:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:109950K .......... .......... .......... .......... .......... 65% 20.4M 2s 2024-04-25T03:24:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110000K .......... .......... .......... .......... .......... 65% 33.9M 2s 2024-04-25T03:24:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110050K .......... .......... .......... .......... .......... 65% 33.8M 2s 2024-04-25T03:24:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110100K .......... .......... .......... .......... .......... 65% 19.6M 2s 2024-04-25T03:24:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110150K .......... .......... .......... .......... .......... 65% 53.5M 2s 2024-04-25T03:24:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110200K .......... .......... .......... .......... .......... 65% 23.1M 2s 2024-04-25T03:24:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110250K .......... .......... .......... .......... .......... 65% 35.6M 2s 2024-04-25T03:24:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110300K .......... .......... .......... .......... .......... 65% 26.8M 2s 2024-04-25T03:24:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110350K .......... .......... .......... .......... .......... 65% 21.1M 2s 2024-04-25T03:24:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110400K .......... .......... .......... .......... .......... 65% 40.8M 2s 2024-04-25T03:24:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110450K .......... .......... .......... .......... .......... 65% 28.8M 2s 2024-04-25T03:24:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110500K .......... .......... .......... .......... .......... 65% 31.8M 2s 2024-04-25T03:24:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110550K .......... .......... .......... .......... .......... 65% 25.8M 2s 2024-04-25T03:24:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110600K .......... .......... .......... .......... .......... 65% 23.1M 2s 2024-04-25T03:24:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110650K .......... .......... .......... .......... .......... 65% 46.3M 2s 2024-04-25T03:24:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110700K .......... .......... .......... .......... .......... 65% 25.4M 2s 2024-04-25T03:24:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110750K .......... .......... .......... .......... .......... 65% 24.6M 2s 2024-04-25T03:24:20.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110800K .......... .......... .......... .......... .......... 65% 34.2M 2s 2024-04-25T03:24:20.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110850K .......... .......... .......... .......... .......... 65% 32.5M 2s 2024-04-25T03:24:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110900K .......... .......... .......... .......... .......... 65% 39.5M 2s 2024-04-25T03:24:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:110950K .......... .......... .......... .......... .......... 65% 22.8M 2s 2024-04-25T03:24:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111000K .......... .......... .......... .......... .......... 65% 43.8M 2s 2024-04-25T03:24:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111050K .......... .......... .......... .......... .......... 65% 23.1M 2s 2024-04-25T03:24:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111100K .......... .......... .......... .......... .......... 65% 46.8M 2s 2024-04-25T03:24:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111150K .......... .......... .......... .......... .......... 65% 15.1M 2s 2024-04-25T03:24:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111200K .......... .......... .......... .......... .......... 65% 54.0M 2s 2024-04-25T03:24:20.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111250K .......... .......... .......... .......... .......... 66% 22.2M 2s 2024-04-25T03:24:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111300K .......... .......... .......... .......... .......... 66% 42.2M 2s 2024-04-25T03:24:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111350K .......... .......... .......... .......... .......... 66% 22.6M 2s 2024-04-25T03:24:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111400K .......... .......... .......... .......... .......... 66% 46.9M 2s 2024-04-25T03:24:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111450K .......... .......... .......... .......... .......... 66% 25.8M 2s 2024-04-25T03:24:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111500K .......... .......... .......... .......... .......... 66% 35.7M 2s 2024-04-25T03:24:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111550K .......... .......... .......... .......... .......... 66% 24.1M 2s 2024-04-25T03:24:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111600K .......... .......... .......... .......... .......... 66% 24.2M 2s 2024-04-25T03:24:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111650K .......... .......... .......... .......... .......... 66% 50.2M 2s 2024-04-25T03:24:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111700K .......... .......... .......... .......... .......... 66% 23.9M 2s 2024-04-25T03:24:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111750K .......... .......... .......... .......... .......... 66% 32.4M 2s 2024-04-25T03:24:20.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111800K .......... .......... .......... .......... .......... 66% 20.9M 2s 2024-04-25T03:24:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111850K .......... .......... .......... .......... .......... 66% 33.0M 2s 2024-04-25T03:24:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111900K .......... .......... .......... .......... .......... 66% 42.6M 2s 2024-04-25T03:24:20.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:111950K .......... .......... .......... .......... .......... 66% 24.8M 2s 2024-04-25T03:24:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112000K .......... .......... .......... .......... .......... 66% 23.6M 2s 2024-04-25T03:24:20.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112050K .......... .......... .......... .......... .......... 66% 36.5M 2s 2024-04-25T03:24:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112100K .......... .......... .......... .......... .......... 66% 30.8M 2s 2024-04-25T03:24:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112150K .......... .......... .......... .......... .......... 66% 29.7M 2s 2024-04-25T03:24:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112200K .......... .......... .......... .......... .......... 66% 27.3M 2s 2024-04-25T03:24:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112250K .......... .......... .......... .......... .......... 66% 30.4M 2s 2024-04-25T03:24:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112300K .......... .......... .......... .......... .......... 66% 21.8M 2s 2024-04-25T03:24:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112350K .......... .......... .......... .......... .......... 66% 26.1M 2s 2024-04-25T03:24:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112400K .......... .......... .......... .......... .......... 66% 23.8M 2s 2024-04-25T03:24:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112450K .......... .......... .......... .......... .......... 66% 40.0M 2s 2024-04-25T03:24:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112500K .......... .......... .......... .......... .......... 66% 28.5M 2s 2024-04-25T03:24:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112550K .......... .......... .......... .......... .......... 66% 27.1M 2s 2024-04-25T03:24:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112600K .......... .......... .......... .......... .......... 66% 37.3M 2s 2024-04-25T03:24:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112650K .......... .......... .......... .......... .......... 66% 39.6M 2s 2024-04-25T03:24:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112700K .......... .......... .......... .......... .......... 66% 22.3M 2s 2024-04-25T03:24:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112750K .......... .......... .......... .......... .......... 66% 26.9M 2s 2024-04-25T03:24:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112800K .......... .......... .......... .......... .......... 66% 36.1M 2s 2024-04-25T03:24:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112850K .......... .......... .......... .......... .......... 66% 21.3M 2s 2024-04-25T03:24:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112900K .......... .......... .......... .......... .......... 67% 25.9M 2s 2024-04-25T03:24:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:112950K .......... .......... .......... .......... .......... 67% 75.1M 2s 2024-04-25T03:24:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113000K .......... .......... .......... .......... .......... 67% 20.3M 2s 2024-04-25T03:24:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113050K .......... .......... .......... .......... .......... 67% 47.7M 2s 2024-04-25T03:24:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113100K .......... .......... .......... .......... .......... 67% 22.4M 2s 2024-04-25T03:24:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113150K .......... .......... .......... .......... .......... 67% 23.4M 2s 2024-04-25T03:24:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113200K .......... .......... .......... .......... .......... 67% 49.6M 2s 2024-04-25T03:24:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113250K .......... .......... .......... .......... .......... 67% 28.0M 2s 2024-04-25T03:24:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113300K .......... .......... .......... .......... .......... 67% 28.8M 2s 2024-04-25T03:24:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113350K .......... .......... .......... .......... .......... 67% 26.5M 2s 2024-04-25T03:24:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113400K .......... .......... .......... .......... .......... 67% 22.4M 2s 2024-04-25T03:24:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113450K .......... .......... .......... .......... .......... 67% 55.3M 2s 2024-04-25T03:24:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113500K .......... .......... .......... .......... .......... 67% 21.5M 2s 2024-04-25T03:24:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113550K .......... .......... .......... .......... .......... 67% 26.4M 2s 2024-04-25T03:24:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113600K .......... .......... .......... .......... .......... 67% 27.5M 2s 2024-04-25T03:24:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113650K .......... .......... .......... .......... .......... 67% 32.5M 2s 2024-04-25T03:24:20.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113700K .......... .......... .......... .......... .......... 67% 46.7M 2s 2024-04-25T03:24:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113750K .......... .......... .......... .......... .......... 67% 23.4M 2s 2024-04-25T03:24:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113800K .......... .......... .......... .......... .......... 67% 24.7M 2s 2024-04-25T03:24:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113850K .......... .......... .......... .......... .......... 67% 73.1M 2s 2024-04-25T03:24:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113900K .......... .......... .......... .......... .......... 67% 35.0M 2s 2024-04-25T03:24:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:113950K .......... .......... .......... .......... .......... 67% 17.8M 2s 2024-04-25T03:24:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114000K .......... .......... .......... .......... .......... 67% 28.5M 2s 2024-04-25T03:24:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114050K .......... .......... .......... .......... .......... 67% 25.4M 2s 2024-04-25T03:24:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114100K .......... .......... .......... .......... .......... 67% 35.0M 2s 2024-04-25T03:24:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114150K .......... .......... .......... .......... .......... 67% 35.3M 2s 2024-04-25T03:24:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114200K .......... .......... .......... .......... .......... 67% 39.1M 2s 2024-04-25T03:24:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114250K .......... .......... .......... .......... .......... 67% 22.2M 2s 2024-04-25T03:24:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114300K .......... .......... .......... .......... .......... 67% 41.2M 2s 2024-04-25T03:24:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114350K .......... .......... .......... .......... .......... 67% 22.4M 2s 2024-04-25T03:24:20.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114400K .......... .......... .......... .......... .......... 67% 24.7M 2s 2024-04-25T03:24:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114450K .......... .......... .......... .......... .......... 67% 41.6M 2s 2024-04-25T03:24:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114500K .......... .......... .......... .......... .......... 67% 25.5M 2s 2024-04-25T03:24:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114550K .......... .......... .......... .......... .......... 67% 37.9M 2s 2024-04-25T03:24:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114600K .......... .......... .......... .......... .......... 68% 20.7M 2s 2024-04-25T03:24:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114650K .......... .......... .......... .......... .......... 68% 26.4M 2s 2024-04-25T03:24:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114700K .......... .......... .......... .......... .......... 68% 38.5M 2s 2024-04-25T03:24:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114750K .......... .......... .......... .......... .......... 68% 23.3M 2s 2024-04-25T03:24:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114800K .......... .......... .......... .......... .......... 68% 29.3M 2s 2024-04-25T03:24:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114850K .......... .......... .......... .......... .......... 68% 32.8M 2s 2024-04-25T03:24:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114900K .......... .......... .......... .......... .......... 68% 26.1M 2s 2024-04-25T03:24:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:114950K .......... .......... .......... .......... .......... 68% 49.1M 2s 2024-04-25T03:24:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115000K .......... .......... .......... .......... .......... 68% 23.5M 2s 2024-04-25T03:24:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115050K .......... .......... .......... .......... .......... 68% 23.5M 2s 2024-04-25T03:24:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115100K .......... .......... .......... .......... .......... 68% 37.9M 2s 2024-04-25T03:24:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115150K .......... .......... .......... .......... .......... 68% 21.5M 2s 2024-04-25T03:24:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115200K .......... .......... .......... .......... .......... 68% 26.7M 2s 2024-04-25T03:24:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115250K .......... .......... .......... .......... .......... 68% 48.2M 2s 2024-04-25T03:24:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115300K .......... .......... .......... .......... .......... 68% 25.3M 2s 2024-04-25T03:24:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115350K .......... .......... .......... .......... .......... 68% 33.8M 2s 2024-04-25T03:24:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115400K .......... .......... .......... .......... .......... 68% 26.1M 2s 2024-04-25T03:24:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115450K .......... .......... .......... .......... .......... 68% 36.9M 2s 2024-04-25T03:24:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115500K .......... .......... .......... .......... .......... 68% 28.9M 2s 2024-04-25T03:24:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115550K .......... .......... .......... .......... .......... 68% 23.7M 2s 2024-04-25T03:24:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115600K .......... .......... .......... .......... .......... 68% 39.4M 2s 2024-04-25T03:24:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115650K .......... .......... .......... .......... .......... 68% 22.6M 2s 2024-04-25T03:24:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115700K .......... .......... .......... .......... .......... 68% 25.1M 2s 2024-04-25T03:24:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115750K .......... .......... .......... .......... .......... 68% 40.4M 2s 2024-04-25T03:24:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115800K .......... .......... .......... .......... .......... 68% 26.0M 2s 2024-04-25T03:24:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115850K .......... .......... .......... .......... .......... 68% 39.3M 2s 2024-04-25T03:24:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115900K .......... .......... .......... .......... .......... 68% 30.0M 2s 2024-04-25T03:24:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:115950K .......... .......... .......... .......... .......... 68% 19.9M 2s 2024-04-25T03:24:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116000K .......... .......... .......... .......... .......... 68% 47.1M 2s 2024-04-25T03:24:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116050K .......... .......... .......... .......... .......... 68% 24.5M 2s 2024-04-25T03:24:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116100K .......... .......... .......... .......... .......... 68% 39.7M 2s 2024-04-25T03:24:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116150K .......... .......... .......... .......... .......... 68% 24.6M 2s 2024-04-25T03:24:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116200K .......... .......... .......... .......... .......... 68% 40.5M 2s 2024-04-25T03:24:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116250K .......... .......... .......... .......... .......... 68% 24.1M 2s 2024-04-25T03:24:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116300K .......... .......... .......... .......... .......... 69% 23.1M 2s 2024-04-25T03:24:20.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116350K .......... .......... .......... .......... .......... 69% 24.3M 2s 2024-04-25T03:24:20.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116400K .......... .......... .......... .......... .......... 69% 40.7M 2s 2024-04-25T03:24:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116450K .......... .......... .......... .......... .......... 69% 21.3M 2s 2024-04-25T03:24:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116500K .......... .......... .......... .......... .......... 69% 52.6M 2s 2024-04-25T03:24:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116550K .......... .......... .......... .......... .......... 69% 23.9M 2s 2024-04-25T03:24:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116600K .......... .......... .......... .......... .......... 69% 24.5M 2s 2024-04-25T03:24:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116650K .......... .......... .......... .......... .......... 69% 33.7M 2s 2024-04-25T03:24:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116700K .......... .......... .......... .......... .......... 69% 60.6M 2s 2024-04-25T03:24:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116750K .......... .......... .......... .......... .......... 69% 20.2M 2s 2024-04-25T03:24:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116800K .......... .......... .......... .......... .......... 69% 22.8M 2s 2024-04-25T03:24:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116850K .......... .......... .......... .......... .......... 69% 27.8M 2s 2024-04-25T03:24:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116900K .......... .......... .......... .......... .......... 69% 35.3M 2s 2024-04-25T03:24:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:116950K .......... .......... .......... .......... .......... 69% 23.7M 2s 2024-04-25T03:24:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117000K .......... .......... .......... .......... .......... 69% 56.9M 2s 2024-04-25T03:24:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117050K .......... .......... .......... .......... .......... 69% 24.3M 2s 2024-04-25T03:24:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117100K .......... .......... .......... .......... .......... 69% 31.8M 2s 2024-04-25T03:24:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117150K .......... .......... .......... .......... .......... 69% 27.4M 2s 2024-04-25T03:24:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117200K .......... .......... .......... .......... .......... 69% 23.6M 2s 2024-04-25T03:24:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117250K .......... .......... .......... .......... .......... 69% 40.3M 2s 2024-04-25T03:24:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117300K .......... .......... .......... .......... .......... 69% 28.7M 2s 2024-04-25T03:24:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117350K .......... .......... .......... .......... .......... 69% 36.6M 2s 2024-04-25T03:24:20.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117400K .......... .......... .......... .......... .......... 69% 24.2M 2s 2024-04-25T03:24:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117450K .......... .......... .......... .......... .......... 69% 20.5M 2s 2024-04-25T03:24:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117500K .......... .......... .......... .......... .......... 69% 60.1M 2s 2024-04-25T03:24:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117550K .......... .......... .......... .......... .......... 69% 21.9M 2s 2024-04-25T03:24:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117600K .......... .......... .......... .......... .......... 69% 24.7M 2s 2024-04-25T03:24:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117650K .......... .......... .......... .......... .......... 69% 41.7M 2s 2024-04-25T03:24:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117700K .......... .......... .......... .......... .......... 69% 25.4M 2s 2024-04-25T03:24:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117750K .......... .......... .......... .......... .......... 69% 44.4M 2s 2024-04-25T03:24:20.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117800K .......... .......... .......... .......... .......... 69% 23.8M 2s 2024-04-25T03:24:20.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117850K .......... .......... .......... .......... .......... 69% 37.1M 2s 2024-04-25T03:24:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117900K .......... .......... .......... .......... .......... 69% 25.0M 2s 2024-04-25T03:24:20.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:117950K .......... .......... .......... .......... .......... 70% 22.9M 2s 2024-04-25T03:24:20.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118000K .......... .......... .......... .......... .......... 70% 26.5M 2s 2024-04-25T03:24:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118050K .......... .......... .......... .......... .......... 70% 42.3M 2s 2024-04-25T03:24:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118100K .......... .......... .......... .......... .......... 70% 23.9M 2s 2024-04-25T03:24:20.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118150K .......... .......... .......... .......... .......... 70% 38.8M 2s 2024-04-25T03:24:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118200K .......... .......... .......... .......... .......... 70% 27.2M 2s 2024-04-25T03:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118250K .......... .......... .......... .......... .......... 70% 43.2M 2s 2024-04-25T03:24:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118300K .......... .......... .......... .......... .......... 70% 22.9M 2s 2024-04-25T03:24:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118350K .......... .......... .......... .......... .......... 70% 23.8M 2s 2024-04-25T03:24:20.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118400K .......... .......... .......... .......... .......... 70% 30.6M 2s 2024-04-25T03:24:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118450K .......... .......... .......... .......... .......... 70% 27.5M 2s 2024-04-25T03:24:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118500K .......... .......... .......... .......... .......... 70% 24.8M 2s 2024-04-25T03:24:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118550K .......... .......... .......... .......... .......... 70% 49.8M 2s 2024-04-25T03:24:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118600K .......... .......... .......... .......... .......... 70% 36.2M 2s 2024-04-25T03:24:20.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118650K .......... .......... .......... .......... .......... 70% 22.3M 2s 2024-04-25T03:24:20.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118700K .......... .......... .......... .......... .......... 70% 27.1M 2s 2024-04-25T03:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118750K .......... .......... .......... .......... .......... 70% 23.0M 2s 2024-04-25T03:24:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118800K .......... .......... .......... .......... .......... 70% 46.6M 2s 2024-04-25T03:24:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118850K .......... .......... .......... .......... .......... 70% 23.7M 2s 2024-04-25T03:24:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118900K .......... .......... .......... .......... .......... 70% 43.2M 2s 2024-04-25T03:24:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:118950K .......... .......... .......... .......... .......... 70% 22.7M 2s 2024-04-25T03:24:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119000K .......... .......... .......... .......... .......... 70% 50.9M 2s 2024-04-25T03:24:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119050K .......... .......... .......... .......... .......... 70% 24.5M 2s 2024-04-25T03:24:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119100K .......... .......... .......... .......... .......... 70% 33.0M 2s 2024-04-25T03:24:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119150K .......... .......... .......... .......... .......... 70% 23.5M 2s 2024-04-25T03:24:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119200K .......... .......... .......... .......... .......... 70% 26.8M 2s 2024-04-25T03:24:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119250K .......... .......... .......... .......... .......... 70% 24.9M 2s 2024-04-25T03:24:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119300K .......... .......... .......... .......... .......... 70% 52.2M 2s 2024-04-25T03:24:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119350K .......... .......... .......... .......... .......... 70% 34.1M 2s 2024-04-25T03:24:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119400K .......... .......... .......... .......... .......... 70% 19.5M 2s 2024-04-25T03:24:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119450K .......... .......... .......... .......... .......... 70% 33.2M 2s 2024-04-25T03:24:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119500K .......... .......... .......... .......... .......... 70% 48.3M 2s 2024-04-25T03:24:20.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119550K .......... .......... .......... .......... .......... 70% 20.8M 2s 2024-04-25T03:24:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119600K .......... .......... .......... .......... .......... 70% 30.6M 2s 2024-04-25T03:24:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119650K .......... .......... .......... .......... .......... 71% 28.6M 2s 2024-04-25T03:24:20.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119700K .......... .......... .......... .......... .......... 71% 24.4M 2s 2024-04-25T03:24:20.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119750K .......... .......... .......... .......... .......... 71% 27.1M 2s 2024-04-25T03:24:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119800K .......... .......... .......... .......... .......... 71% 46.5M 2s 2024-04-25T03:24:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119850K .......... .......... .......... .......... .......... 71% 24.6M 2s 2024-04-25T03:24:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119900K .......... .......... .......... .......... .......... 71% 32.7M 2s 2024-04-25T03:24:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:119950K .......... .......... .......... .......... .......... 71% 26.9M 2s 2024-04-25T03:24:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120000K .......... .......... .......... .......... .......... 71% 24.5M 2s 2024-04-25T03:24:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120050K .......... .......... .......... .......... .......... 71% 42.3M 2s 2024-04-25T03:24:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120100K .......... .......... .......... .......... .......... 71% 24.2M 2s 2024-04-25T03:24:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120150K .......... .......... .......... .......... .......... 71% 36.1M 2s 2024-04-25T03:24:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120200K .......... .......... .......... .......... .......... 71% 21.6M 2s 2024-04-25T03:24:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120250K .......... .......... .......... .......... .......... 71% 27.3M 2s 2024-04-25T03:24:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120300K .......... .......... .......... .......... .......... 71% 40.1M 2s 2024-04-25T03:24:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120350K .......... .......... .......... .......... .......... 71% 25.6M 2s 2024-04-25T03:24:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120400K .......... .......... .......... .......... .......... 71% 24.6M 2s 2024-04-25T03:24:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120450K .......... .......... .......... .......... .......... 71% 39.2M 2s 2024-04-25T03:24:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120500K .......... .......... .......... .......... .......... 71% 23.0M 2s 2024-04-25T03:24:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120550K .......... .......... .......... .......... .......... 71% 51.6M 2s 2024-04-25T03:24:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120600K .......... .......... .......... .......... .......... 71% 22.9M 2s 2024-04-25T03:24:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120650K .......... .......... .......... .......... .......... 71% 34.5M 2s 2024-04-25T03:24:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120700K .......... .......... .......... .......... .......... 71% 26.9M 2s 2024-04-25T03:24:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120750K .......... .......... .......... .......... .......... 71% 22.0M 2s 2024-04-25T03:24:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120800K .......... .......... .......... .......... .......... 71% 22.5M 2s 2024-04-25T03:24:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120850K .......... .......... .......... .......... .......... 71% 37.6M 2s 2024-04-25T03:24:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120900K .......... .......... .......... .......... .......... 71% 24.1M 2s 2024-04-25T03:24:20.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:120950K .......... .......... .......... .......... .......... 71% 27.7M 2s 2024-04-25T03:24:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121000K .......... .......... .......... .......... .......... 71% 24.1M 2s 2024-04-25T03:24:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121050K .......... .......... .......... .......... .......... 71% 47.0M 2s 2024-04-25T03:24:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121100K .......... .......... .......... .......... .......... 71% 22.3M 2s 2024-04-25T03:24:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121150K .......... .......... .......... .......... .......... 71% 24.4M 2s 2024-04-25T03:24:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121200K .......... .......... .......... .......... .......... 71% 43.1M 2s 2024-04-25T03:24:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121250K .......... .......... .......... .......... .......... 71% 19.6M 2s 2024-04-25T03:24:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121300K .......... .......... .......... .......... .......... 71% 23.2M 2s 2024-04-25T03:24:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121350K .......... .......... .......... .......... .......... 72% 44.8M 2s 2024-04-25T03:24:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121400K .......... .......... .......... .......... .......... 72% 36.4M 2s 2024-04-25T03:24:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121450K .......... .......... .......... .......... .......... 72% 24.0M 2s 2024-04-25T03:24:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121500K .......... .......... .......... .......... .......... 72% 24.6M 2s 2024-04-25T03:24:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121550K .......... .......... .......... .......... .......... 72% 21.9M 2s 2024-04-25T03:24:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121600K .......... .......... .......... .......... .......... 72% 53.3M 2s 2024-04-25T03:24:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121650K .......... .......... .......... .......... .......... 72% 22.9M 2s 2024-04-25T03:24:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121700K .......... .......... .......... .......... .......... 72% 39.8M 2s 2024-04-25T03:24:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121750K .......... .......... .......... .......... .......... 72% 22.9M 2s 2024-04-25T03:24:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121800K .......... .......... .......... .......... .......... 72% 56.3M 2s 2024-04-25T03:24:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121850K .......... .......... .......... .......... .......... 72% 23.8M 2s 2024-04-25T03:24:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121900K .......... .......... .......... .......... .......... 72% 30.7M 2s 2024-04-25T03:24:20.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:121950K .......... .......... .......... .......... .......... 72% 27.3M 2s 2024-04-25T03:24:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122000K .......... .......... .......... .......... .......... 72% 24.2M 2s 2024-04-25T03:24:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122050K .......... .......... .......... .......... .......... 72% 23.4M 2s 2024-04-25T03:24:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122100K .......... .......... .......... .......... .......... 72% 40.2M 2s 2024-04-25T03:24:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122150K .......... .......... .......... .......... .......... 72% 27.4M 2s 2024-04-25T03:24:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122200K .......... .......... .......... .......... .......... 72% 39.0M 2s 2024-04-25T03:24:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122250K .......... .......... .......... .......... .......... 72% 24.9M 2s 2024-04-25T03:24:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122300K .......... .......... .......... .......... .......... 72% 45.5M 2s 2024-04-25T03:24:20.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122350K .......... .......... .......... .......... .......... 72% 26.7M 2s 2024-04-25T03:24:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122400K .......... .......... .......... .......... .......... 72% 21.6M 2s 2024-04-25T03:24:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122450K .......... .......... .......... .......... .......... 72% 37.2M 2s 2024-04-25T03:24:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122500K .......... .......... .......... .......... .......... 72% 22.5M 2s 2024-04-25T03:24:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122550K .......... .......... .......... .......... .......... 72% 25.9M 2s 2024-04-25T03:24:20.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122600K .......... .......... .......... .......... .......... 72% 47.5M 2s 2024-04-25T03:24:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122650K .......... .......... .......... .......... .......... 72% 23.9M 2s 2024-04-25T03:24:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122700K .......... .......... .......... .......... .......... 72% 44.0M 2s 2024-04-25T03:24:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122750K .......... .......... .......... .......... .......... 72% 23.1M 2s 2024-04-25T03:24:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122800K .......... .......... .......... .......... .......... 72% 25.4M 2s 2024-04-25T03:24:20.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122850K .......... .......... .......... .......... .......... 72% 45.8M 2s 2024-04-25T03:24:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122900K .......... .......... .......... .......... .......... 72% 26.4M 2s 2024-04-25T03:24:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:122950K .......... .......... .......... .......... .......... 72% 32.7M 2s 2024-04-25T03:24:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123000K .......... .......... .......... .......... .......... 72% 24.9M 2s 2024-04-25T03:24:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123050K .......... .......... .......... .......... .......... 73% 21.5M 2s 2024-04-25T03:24:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123100K .......... .......... .......... .......... .......... 73% 63.4M 2s 2024-04-25T03:24:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123150K .......... .......... .......... .......... .......... 73% 22.6M 2s 2024-04-25T03:24:20.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123200K .......... .......... .......... .......... .......... 73% 24.1M 2s 2024-04-25T03:24:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123250K .......... .......... .......... .......... .......... 73% 35.4M 2s 2024-04-25T03:24:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123300K .......... .......... .......... .......... .......... 73% 23.8M 2s 2024-04-25T03:24:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123350K .......... .......... .......... .......... .......... 73% 29.4M 2s 2024-04-25T03:24:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123400K .......... .......... .......... .......... .......... 73% 28.5M 2s 2024-04-25T03:24:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123450K .......... .......... .......... .......... .......... 73% 40.3M 2s 2024-04-25T03:24:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123500K .......... .......... .......... .......... .......... 73% 24.9M 2s 2024-04-25T03:24:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123550K .......... .......... .......... .......... .......... 73% 21.1M 2s 2024-04-25T03:24:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123600K .......... .......... .......... .......... .......... 73% 22.1M 2s 2024-04-25T03:24:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123650K .......... .......... .......... .......... .......... 73% 57.6M 2s 2024-04-25T03:24:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123700K .......... .......... .......... .......... .......... 73% 21.1M 2s 2024-04-25T03:24:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123750K .......... .......... .......... .......... .......... 73% 43.6M 2s 2024-04-25T03:24:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123800K .......... .......... .......... .......... .......... 73% 24.7M 2s 2024-04-25T03:24:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123850K .......... .......... .......... .......... .......... 73% 46.9M 2s 2024-04-25T03:24:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123900K .......... .......... .......... .......... .......... 73% 24.5M 2s 2024-04-25T03:24:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:123950K .......... .......... .......... .......... .......... 73% 22.7M 2s 2024-04-25T03:24:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124000K .......... .......... .......... .......... .......... 73% 35.2M 2s 2024-04-25T03:24:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124050K .......... .......... .......... .......... .......... 73% 31.7M 2s 2024-04-25T03:24:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124100K .......... .......... .......... .......... .......... 73% 39.8M 2s 2024-04-25T03:24:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124150K .......... .......... .......... .......... .......... 73% 22.3M 2s 2024-04-25T03:24:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124200K .......... .......... .......... .......... .......... 73% 25.7M 2s 2024-04-25T03:24:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124250K .......... .......... .......... .......... .......... 73% 29.4M 2s 2024-04-25T03:24:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124300K .......... .......... .......... .......... .......... 73% 27.8M 2s 2024-04-25T03:24:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124350K .......... .......... .......... .......... .......... 73% 23.8M 2s 2024-04-25T03:24:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124400K .......... .......... .......... .......... .......... 73% 47.5M 2s 2024-04-25T03:24:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124450K .......... .......... .......... .......... .......... 73% 23.1M 2s 2024-04-25T03:24:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124500K .......... .......... .......... .......... .......... 73% 25.7M 2s 2024-04-25T03:24:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124550K .......... .......... .......... .......... .......... 73% 34.7M 2s 2024-04-25T03:24:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124600K .......... .......... .......... .......... .......... 73% 57.0M 2s 2024-04-25T03:24:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124650K .......... .......... .......... .......... .......... 73% 21.7M 2s 2024-04-25T03:24:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124700K .......... .......... .......... .......... .......... 74% 26.5M 2s 2024-04-25T03:24:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124750K .......... .......... .......... .......... .......... 74% 24.3M 2s 2024-04-25T03:24:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124800K .......... .......... .......... .......... .......... 74% 31.5M 2s 2024-04-25T03:24:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124850K .......... .......... .......... .......... .......... 74% 25.2M 2s 2024-04-25T03:24:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124900K .......... .......... .......... .......... .......... 74% 41.1M 2s 2024-04-25T03:24:20.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:124950K .......... .......... .......... .......... .......... 74% 24.4M 2s 2024-04-25T03:24:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125000K .......... .......... .......... .......... .......... 74% 43.9M 2s 2024-04-25T03:24:20.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125050K .......... .......... .......... .......... .......... 74% 22.2M 2s 2024-04-25T03:24:20.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125100K .......... .......... .......... .......... .......... 74% 43.2M 2s 2024-04-25T03:24:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125150K .......... .......... .......... .......... .......... 74% 26.5M 2s 2024-04-25T03:24:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125200K .......... .......... .......... .......... .......... 74% 24.3M 2s 2024-04-25T03:24:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125250K .......... .......... .......... .......... .......... 74% 35.4M 2s 2024-04-25T03:24:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125300K .......... .......... .......... .......... .......... 74% 20.9M 2s 2024-04-25T03:24:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125350K .......... .......... .......... .......... .......... 74% 25.1M 2s 2024-04-25T03:24:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125400K .......... .......... .......... .......... .......... 74% 70.7M 2s 2024-04-25T03:24:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125450K .......... .......... .......... .......... .......... 74% 22.8M 2s 2024-04-25T03:24:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125500K .......... .......... .......... .......... .......... 74% 31.7M 2s 2024-04-25T03:24:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125550K .......... .......... .......... .......... .......... 74% 29.3M 2s 2024-04-25T03:24:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125600K .......... .......... .......... .......... .......... 74% 22.6M 2s 2024-04-25T03:24:20.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125650K .......... .......... .......... .......... .......... 74% 50.8M 2s 2024-04-25T03:24:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125700K .......... .......... .......... .......... .......... 74% 28.4M 2s 2024-04-25T03:24:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125750K .......... .......... .......... .......... .......... 74% 19.2M 2s 2024-04-25T03:24:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125800K .......... .......... .......... .......... .......... 74% 66.1M 2s 2024-04-25T03:24:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125850K .......... .......... .......... .......... .......... 74% 24.2M 2s 2024-04-25T03:24:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125900K .......... .......... .......... .......... .......... 74% 41.0M 2s 2024-04-25T03:24:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:125950K .......... .......... .......... .......... .......... 74% 21.8M 2s 2024-04-25T03:24:20.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126000K .......... .......... .......... .......... .......... 74% 22.6M 2s 2024-04-25T03:24:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126050K .......... .......... .......... .......... .......... 74% 36.8M 2s 2024-04-25T03:24:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126100K .......... .......... .......... .......... .......... 74% 24.6M 2s 2024-04-25T03:24:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126150K .......... .......... .......... .......... .......... 74% 44.9M 2s 2024-04-25T03:24:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126200K .......... .......... .......... .......... .......... 74% 24.3M 2s 2024-04-25T03:24:20.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126250K .......... .......... .......... .......... .......... 74% 38.9M 2s 2024-04-25T03:24:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126300K .......... .......... .......... .......... .......... 74% 24.1M 2s 2024-04-25T03:24:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126350K .......... .......... .......... .......... .......... 74% 24.9M 2s 2024-04-25T03:24:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126400K .......... .......... .......... .......... .......... 75% 22.0M 2s 2024-04-25T03:24:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126450K .......... .......... .......... .......... .......... 75% 57.2M 2s 2024-04-25T03:24:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126500K .......... .......... .......... .......... .......... 75% 23.6M 2s 2024-04-25T03:24:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126550K .......... .......... .......... .......... .......... 75% 28.8M 2s 2024-04-25T03:24:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126600K .......... .......... .......... .......... .......... 75% 26.6M 2s 2024-04-25T03:24:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126650K .......... .......... .......... .......... .......... 75% 38.8M 2s 2024-04-25T03:24:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126700K .......... .......... .......... .......... .......... 75% 27.0M 2s 2024-04-25T03:24:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126750K .......... .......... .......... .......... .......... 75% 23.1M 2s 2024-04-25T03:24:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126800K .......... .......... .......... .......... .......... 75% 39.4M 2s 2024-04-25T03:24:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126850K .......... .......... .......... .......... .......... 75% 25.5M 1s 2024-04-25T03:24:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126900K .......... .......... .......... .......... .......... 75% 11.3M 1s 2024-04-25T03:24:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:126950K .......... .......... .......... .......... .......... 75% 4.61M 2s 2024-04-25T03:24:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127000K .......... .......... .......... .......... .......... 75% 38.2M 1s 2024-04-25T03:24:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127050K .......... .......... .......... .......... .......... 75% 22.8M 1s 2024-04-25T03:24:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127100K .......... .......... .......... .......... .......... 75% 683K 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127150K .......... .......... .......... .......... .......... 75% 150M 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127200K .......... .......... .......... .......... .......... 75% 182M 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127250K .......... .......... .......... .......... .......... 75% 188M 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127300K .......... .......... .......... .......... .......... 75% 176M 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127350K .......... .......... .......... .......... .......... 75% 165M 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127400K .......... .......... .......... .......... .......... 75% 178M 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127450K .......... .......... .......... .......... .......... 75% 172M 2s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127500K .......... .......... .......... .......... .......... 75% 168M 1s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127550K .......... .......... .......... .......... .......... 75% 159M 1s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127600K .......... .......... .......... .......... .......... 75% 184M 1s 2024-04-25T03:24:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127650K .......... .......... .......... .......... .......... 75% 6.03M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127700K .......... .......... .......... .......... .......... 75% 148M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127750K .......... .......... .......... .......... .......... 75% 167M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127800K .......... .......... .......... .......... .......... 75% 188M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127850K .......... .......... .......... .......... .......... 75% 172M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127900K .......... .......... .......... .......... .......... 75% 165M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:127950K .......... .......... .......... .......... .......... 75% 153M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128000K .......... .......... .......... .......... .......... 75% 178M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128050K .......... .......... .......... .......... .......... 75% 193M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128100K .......... .......... .......... .......... .......... 76% 185M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128150K .......... .......... .......... .......... .......... 76% 170M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128200K .......... .......... .......... .......... .......... 76% 177M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128250K .......... .......... .......... .......... .......... 76% 180M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128300K .......... .......... .......... .......... .......... 76% 190M 1s 2024-04-25T03:24:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128350K .......... .......... .......... .......... .......... 76% 154M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128400K .......... .......... .......... .......... .......... 76% 182M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128450K .......... .......... .......... .......... .......... 76% 186M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128500K .......... .......... .......... .......... .......... 76% 166M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128550K .......... .......... .......... .......... .......... 76% 88.6M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128600K .......... .......... .......... .......... .......... 76% 196M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128650K .......... .......... .......... .......... .......... 76% 187M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128700K .......... .......... .......... .......... .......... 76% 170M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128750K .......... .......... .......... .......... .......... 76% 157M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128800K .......... .......... .......... .......... .......... 76% 183M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128850K .......... .......... .......... .......... .......... 76% 177M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128900K .......... .......... .......... .......... .......... 76% 175M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:128950K .......... .......... .......... .......... .......... 76% 158M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129000K .......... .......... .......... .......... .......... 76% 179M 1s 2024-04-25T03:24:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129050K .......... .......... .......... .......... .......... 76% 173M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129100K .......... .......... .......... .......... .......... 76% 186M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129150K .......... .......... .......... .......... .......... 76% 152M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129200K .......... .......... .......... .......... .......... 76% 175M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129250K .......... .......... .......... .......... .......... 76% 180M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129300K .......... .......... .......... .......... .......... 76% 188M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129350K .......... .......... .......... .......... .......... 76% 176M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129400K .......... .......... .......... .......... .......... 76% 179M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129450K .......... .......... .......... .......... .......... 76% 181M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129500K .......... .......... .......... .......... .......... 76% 169M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129550K .......... .......... .......... .......... .......... 76% 147M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129600K .......... .......... .......... .......... .......... 76% 174M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129650K .......... .......... .......... .......... .......... 76% 204M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129700K .......... .......... .......... .......... .......... 76% 199M 1s 2024-04-25T03:24:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129750K .......... .......... .......... .......... .......... 77% 152M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129800K .......... .......... .......... .......... .......... 77% 182M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129850K .......... .......... .......... .......... .......... 77% 199M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129900K .......... .......... .......... .......... .......... 77% 39.1M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:129950K .......... .......... .......... .......... .......... 77% 41.9M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130000K .......... .......... .......... .......... .......... 77% 26.7M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130050K .......... .......... .......... .......... .......... 77% 23.0M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130100K .......... .......... .......... .......... .......... 77% 23.4M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130150K .......... .......... .......... .......... .......... 77% 153M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130200K .......... .......... .......... .......... .......... 77% 27.0M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130250K .......... .......... .......... .......... .......... 77% 33.7M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130300K .......... .......... .......... .......... .......... 77% 19.9M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130350K .......... .......... .......... .......... .......... 77% 17.1M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130400K .......... .......... .......... .......... .......... 77% 128M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130450K .......... .......... .......... .......... .......... 77% 27.8M 1s 2024-04-25T03:24:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130500K .......... .......... .......... .......... .......... 77% 24.2M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130550K .......... .......... .......... .......... .......... 77% 20.2M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130600K .......... .......... .......... .......... .......... 77% 15.6M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130650K .......... .......... .......... .......... .......... 77% 28.9M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130700K .......... .......... .......... .......... .......... 77% 175M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130750K .......... .......... .......... .......... .......... 77% 23.7M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130800K .......... .......... .......... .......... .......... 77% 22.7M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130850K .......... .......... .......... .......... .......... 77% 23.8M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130900K .......... .......... .......... .......... .......... 77% 128M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:130950K .......... .......... .......... .......... .......... 77% 27.9M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131000K .......... .......... .......... .......... .......... 77% 22.1M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131050K .......... .......... .......... .......... .......... 77% 27.8M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131100K .......... .......... .......... .......... .......... 77% 38.2M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131150K .......... .......... .......... .......... .......... 77% 23.5M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131200K .......... .......... .......... .......... .......... 77% 24.5M 1s 2024-04-25T03:24:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131250K .......... .......... .......... .......... .......... 77% 47.5M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131300K .......... .......... .......... .......... .......... 77% 23.6M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131350K .......... .......... .......... .......... .......... 77% 38.8M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131400K .......... .......... .......... .......... .......... 77% 24.0M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131450K .......... .......... .......... .......... .......... 78% 48.0M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131500K .......... .......... .......... .......... .......... 78% 20.5M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131550K .......... .......... .......... .......... .......... 78% 24.7M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131600K .......... .......... .......... .......... .......... 78% 34.2M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131650K .......... .......... .......... .......... .......... 78% 25.3M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131700K .......... .......... .......... .......... .......... 78% 23.7M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131750K .......... .......... .......... .......... .......... 78% 64.8M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131800K .......... .......... .......... .......... .......... 78% 29.5M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131850K .......... .......... .......... .......... .......... 78% 23.7M 1s 2024-04-25T03:24:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131900K .......... .......... .......... .......... .......... 78% 28.6M 1s 2024-04-25T03:24:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:131950K .......... .......... .......... .......... .......... 78% 20.3M 1s 2024-04-25T03:24:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132000K .......... .......... .......... .......... .......... 78% 51.3M 1s 2024-04-25T03:24:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132050K .......... .......... .......... .......... .......... 78% 30.6M 1s 2024-04-25T03:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132100K .......... .......... .......... .......... .......... 78% 25.6M 1s 2024-04-25T03:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132150K .......... .......... .......... .......... .......... 78% 22.8M 1s 2024-04-25T03:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132200K .......... .......... .......... .......... .......... 78% 75.8M 1s 2024-04-25T03:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132250K .......... .......... .......... .......... .......... 78% 24.3M 1s 2024-04-25T03:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132300K .......... .......... .......... .......... .......... 78% 34.3M 1s 2024-04-25T03:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132350K .......... .......... .......... .......... .......... 78% 22.9M 1s 2024-04-25T03:24:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132400K .......... .......... .......... .......... .......... 78% 22.7M 1s 2024-04-25T03:24:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132450K .......... .......... .......... .......... .......... 78% 29.1M 1s 2024-04-25T03:24:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132500K .......... .......... .......... .......... .......... 78% 43.0M 1s 2024-04-25T03:24:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132550K .......... .......... .......... .......... .......... 78% 22.2M 1s 2024-04-25T03:24:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132600K .......... .......... .......... .......... .......... 78% 37.4M 1s 2024-04-25T03:24:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132650K .......... .......... .......... .......... .......... 78% 25.6M 1s 2024-04-25T03:24:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132700K .......... .......... .......... .......... .......... 78% 51.5M 1s 2024-04-25T03:24:21.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132750K .......... .......... .......... .......... .......... 78% 23.2M 1s 2024-04-25T03:24:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132800K .......... .......... .......... .......... .......... 78% 27.6M 1s 2024-04-25T03:24:21.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132850K .......... .......... .......... .......... .......... 78% 33.5M 1s 2024-04-25T03:24:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132900K .......... .......... .......... .......... .......... 78% 24.0M 1s 2024-04-25T03:24:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:132950K .......... .......... .......... .......... .......... 78% 21.5M 1s 2024-04-25T03:24:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133000K .......... .......... .......... .......... .......... 78% 48.1M 1s 2024-04-25T03:24:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133050K .......... .......... .......... .......... .......... 78% 21.9M 1s 2024-04-25T03:24:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133100K .......... .......... .......... .......... .......... 78% 37.4M 1s 2024-04-25T03:24:21.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133150K .......... .......... .......... .......... .......... 79% 26.2M 1s 2024-04-25T03:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133200K .......... .......... .......... .......... .......... 79% 23.8M 1s 2024-04-25T03:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133250K .......... .......... .......... .......... .......... 79% 41.1M 1s 2024-04-25T03:24:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133300K .......... .......... .......... .......... .......... 79% 34.1M 1s 2024-04-25T03:24:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133350K .......... .......... .......... .......... .......... 79% 24.7M 1s 2024-04-25T03:24:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133400K .......... .......... .......... .......... .......... 79% 24.7M 1s 2024-04-25T03:24:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133450K .......... .......... .......... .......... .......... 79% 23.5M 1s 2024-04-25T03:24:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133500K .......... .......... .......... .......... .......... 79% 51.3M 1s 2024-04-25T03:24:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133550K .......... .......... .......... .......... .......... 79% 22.4M 1s 2024-04-25T03:24:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133600K .......... .......... .......... .......... .......... 79% 27.3M 1s 2024-04-25T03:24:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133650K .......... .......... .......... .......... .......... 79% 33.4M 1s 2024-04-25T03:24:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133700K .......... .......... .......... .......... .......... 79% 25.3M 1s 2024-04-25T03:24:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133750K .......... .......... .......... .......... .......... 79% 53.5M 1s 2024-04-25T03:24:21.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133800K .......... .......... .......... .......... .......... 79% 28.3M 1s 2024-04-25T03:24:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133850K .......... .......... .......... .......... .......... 79% 27.4M 1s 2024-04-25T03:24:21.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133900K .......... .......... .......... .......... .......... 79% 25.1M 1s 2024-04-25T03:24:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:133950K .......... .......... .......... .......... .......... 79% 23.3M 1s 2024-04-25T03:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134000K .......... .......... .......... .......... .......... 79% 23.6M 1s 2024-04-25T03:24:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134050K .......... .......... .......... .......... .......... 79% 60.7M 1s 2024-04-25T03:24:21.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134100K .......... .......... .......... .......... .......... 79% 22.0M 1s 2024-04-25T03:24:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134150K .......... .......... .......... .......... .......... 79% 36.7M 1s 2024-04-25T03:24:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134200K .......... .......... .......... .......... .......... 79% 25.8M 1s 2024-04-25T03:24:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134250K .......... .......... .......... .......... .......... 79% 47.3M 1s 2024-04-25T03:24:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134300K .......... .......... .......... .......... .......... 79% 23.0M 1s 2024-04-25T03:24:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134350K .......... .......... .......... .......... .......... 79% 24.9M 1s 2024-04-25T03:24:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134400K .......... .......... .......... .......... .......... 79% 41.0M 1s 2024-04-25T03:24:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134450K .......... .......... .......... .......... .......... 79% 24.7M 1s 2024-04-25T03:24:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134500K .......... .......... .......... .......... .......... 79% 37.8M 1s 2024-04-25T03:24:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134550K .......... .......... .......... .......... .......... 79% 26.6M 1s 2024-04-25T03:24:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134600K .......... .......... .......... .......... .......... 79% 36.1M 1s 2024-04-25T03:24:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134650K .......... .......... .......... .......... .......... 79% 25.3M 1s 2024-04-25T03:24:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134700K .......... .......... .......... .......... .......... 79% 21.1M 1s 2024-04-25T03:24:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134750K .......... .......... .......... .......... .......... 79% 24.8M 1s 2024-04-25T03:24:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134800K .......... .......... .......... .......... .......... 79% 51.5M 1s 2024-04-25T03:24:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134850K .......... .......... .......... .......... .......... 80% 24.7M 1s 2024-04-25T03:24:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134900K .......... .......... .......... .......... .......... 80% 39.8M 1s 2024-04-25T03:24:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:134950K .......... .......... .......... .......... .......... 80% 24.3M 1s 2024-04-25T03:24:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135000K .......... .......... .......... .......... .......... 80% 46.1M 1s 2024-04-25T03:24:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135050K .......... .......... .......... .......... .......... 80% 24.7M 1s 2024-04-25T03:24:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135100K .......... .......... .......... .......... .......... 80% 37.8M 1s 2024-04-25T03:24:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135150K .......... .......... .......... .......... .......... 80% 22.5M 1s 2024-04-25T03:24:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135200K .......... .......... .......... .......... .......... 80% 23.7M 1s 2024-04-25T03:24:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135250K .......... .......... .......... .......... .......... 80% 24.0M 1s 2024-04-25T03:24:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135300K .......... .......... .......... .......... .......... 80% 42.9M 1s 2024-04-25T03:24:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135350K .......... .......... .......... .......... .......... 80% 23.3M 1s 2024-04-25T03:24:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135400K .......... .......... .......... .......... .......... 80% 44.1M 1s 2024-04-25T03:24:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135450K .......... .......... .......... .......... .......... 80% 24.8M 1s 2024-04-25T03:24:21.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135500K .......... .......... .......... .......... .......... 80% 46.7M 1s 2024-04-25T03:24:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135550K .......... .......... .......... .......... .......... 80% 24.3M 1s 2024-04-25T03:24:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135600K .......... .......... .......... .......... .......... 80% 24.5M 1s 2024-04-25T03:24:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135650K .......... .......... .......... .......... .......... 80% 36.7M 1s 2024-04-25T03:24:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135700K .......... .......... .......... .......... .......... 80% 22.3M 1s 2024-04-25T03:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135750K .......... .......... .......... .......... .......... 80% 26.4M 1s 2024-04-25T03:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135800K .......... .......... .......... .......... .......... 80% 42.8M 1s 2024-04-25T03:24:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135850K .......... .......... .......... .......... .......... 80% 23.8M 1s 2024-04-25T03:24:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135900K .......... .......... .......... .......... .......... 80% 38.8M 1s 2024-04-25T03:24:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:135950K .......... .......... .......... .......... .......... 80% 24.9M 1s 2024-04-25T03:24:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136000K .......... .......... .......... .......... .......... 80% 23.0M 1s 2024-04-25T03:24:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136050K .......... .......... .......... .......... .......... 80% 43.4M 1s 2024-04-25T03:24:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136100K .......... .......... .......... .......... .......... 80% 29.0M 1s 2024-04-25T03:24:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136150K .......... .......... .......... .......... .......... 80% 35.8M 1s 2024-04-25T03:24:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136200K .......... .......... .......... .......... .......... 80% 23.1M 1s 2024-04-25T03:24:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136250K .......... .......... .......... .......... .......... 80% 23.8M 1s 2024-04-25T03:24:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136300K .......... .......... .......... .......... .......... 80% 46.0M 1s 2024-04-25T03:24:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136350K .......... .......... .......... .......... .......... 80% 25.3M 1s 2024-04-25T03:24:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136400K .......... .......... .......... .......... .......... 80% 23.1M 1s 2024-04-25T03:24:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136450K .......... .......... .......... .......... .......... 80% 31.9M 1s 2024-04-25T03:24:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136500K .......... .......... .......... .......... .......... 81% 26.7M 1s 2024-04-25T03:24:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136550K .......... .......... .......... .......... .......... 81% 54.9M 1s 2024-04-25T03:24:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136600K .......... .......... .......... .......... .......... 81% 22.0M 1s 2024-04-25T03:24:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136650K .......... .......... .......... .......... .......... 81% 44.5M 1s 2024-04-25T03:24:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136700K .......... .......... .......... .......... .......... 81% 24.9M 1s 2024-04-25T03:24:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136750K .......... .......... .......... .......... .......... 81% 23.5M 1s 2024-04-25T03:24:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136800K .......... .......... .......... .......... .......... 81% 23.2M 1s 2024-04-25T03:24:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136850K .......... .......... .......... .......... .......... 81% 53.4M 1s 2024-04-25T03:24:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136900K .......... .......... .......... .......... .......... 81% 30.2M 1s 2024-04-25T03:24:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:136950K .......... .......... .......... .......... .......... 81% 31.3M 1s 2024-04-25T03:24:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137000K .......... .......... .......... .......... .......... 81% 17.7M 1s 2024-04-25T03:24:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137050K .......... .......... .......... .......... .......... 81% 61.4M 1s 2024-04-25T03:24:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137100K .......... .......... .......... .......... .......... 81% 27.7M 1s 2024-04-25T03:24:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137150K .......... .......... .......... .......... .......... 81% 21.4M 1s 2024-04-25T03:24:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137200K .......... .......... .......... .......... .......... 81% 24.7M 1s 2024-04-25T03:24:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137250K .......... .......... .......... .......... .......... 81% 36.3M 1s 2024-04-25T03:24:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137300K .......... .......... .......... .......... .......... 81% 43.4M 1s 2024-04-25T03:24:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137350K .......... .......... .......... .......... .......... 81% 22.6M 1s 2024-04-25T03:24:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137400K .......... .......... .......... .......... .......... 81% 40.2M 1s 2024-04-25T03:24:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137450K .......... .......... .......... .......... .......... 81% 25.9M 1s 2024-04-25T03:24:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137500K .......... .......... .......... .......... .......... 81% 47.6M 1s 2024-04-25T03:24:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137550K .......... .......... .......... .......... .......... 81% 22.9M 1s 2024-04-25T03:24:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137600K .......... .......... .......... .......... .......... 81% 9.81M 1s 2024-04-25T03:24:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137650K .......... .......... .......... .......... .......... 81% 146M 1s 2024-04-25T03:24:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137700K .......... .......... .......... .......... .......... 81% 66.5M 1s 2024-04-25T03:24:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137750K .......... .......... .......... .......... .......... 81% 52.0M 1s 2024-04-25T03:24:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137800K .......... .......... .......... .......... .......... 81% 32.7M 1s 2024-04-25T03:24:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137850K .......... .......... .......... .......... .......... 81% 30.7M 1s 2024-04-25T03:24:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137900K .......... .......... .......... .......... .......... 81% 25.0M 1s 2024-04-25T03:24:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:137950K .......... .......... .......... .......... .......... 81% 22.7M 1s 2024-04-25T03:24:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138000K .......... .......... .......... .......... .......... 81% 27.2M 1s 2024-04-25T03:24:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138050K .......... .......... .......... .......... .......... 81% 46.5M 1s 2024-04-25T03:24:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138100K .......... .......... .......... .......... .......... 81% 26.2M 1s 2024-04-25T03:24:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138150K .......... .......... .......... .......... .......... 81% 25.2M 1s 2024-04-25T03:24:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138200K .......... .......... .......... .......... .......... 82% 29.6M 1s 2024-04-25T03:24:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138250K .......... .......... .......... .......... .......... 82% 26.7M 1s 2024-04-25T03:24:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138300K .......... .......... .......... .......... .......... 82% 51.7M 1s 2024-04-25T03:24:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138350K .......... .......... .......... .......... .......... 82% 23.5M 1s 2024-04-25T03:24:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138400K .......... .......... .......... .......... .......... 82% 24.3M 1s 2024-04-25T03:24:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138450K .......... .......... .......... .......... .......... 82% 36.8M 1s 2024-04-25T03:24:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138500K .......... .......... .......... .......... .......... 82% 22.3M 1s 2024-04-25T03:24:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138550K .......... .......... .......... .......... .......... 82% 27.0M 1s 2024-04-25T03:24:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138600K .......... .......... .......... .......... .......... 82% 53.8M 1s 2024-04-25T03:24:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138650K .......... .......... .......... .......... .......... 82% 22.0M 1s 2024-04-25T03:24:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138700K .......... .......... .......... .......... .......... 82% 45.9M 1s 2024-04-25T03:24:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138750K .......... .......... .......... .......... .......... 82% 21.9M 1s 2024-04-25T03:24:21.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138800K .......... .......... .......... .......... .......... 82% 23.6M 1s 2024-04-25T03:24:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138850K .......... .......... .......... .......... .......... 82% 49.9M 1s 2024-04-25T03:24:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138900K .......... .......... .......... .......... .......... 82% 26.1M 1s 2024-04-25T03:24:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:138950K .......... .......... .......... .......... .......... 82% 33.0M 1s 2024-04-25T03:24:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139000K .......... .......... .......... .......... .......... 82% 18.9M 1s 2024-04-25T03:24:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139050K .......... .......... .......... .......... .......... 82% 59.7M 1s 2024-04-25T03:24:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139100K .......... .......... .......... .......... .......... 82% 24.2M 1s 2024-04-25T03:24:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139150K .......... .......... .......... .......... .......... 82% 23.0M 1s 2024-04-25T03:24:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139200K .......... .......... .......... .......... .......... 82% 13.0M 1s 2024-04-25T03:24:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139250K .......... .......... .......... .......... .......... 82% 17.3M 1s 2024-04-25T03:24:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139300K .......... .......... .......... .......... .......... 82% 40.4M 1s 2024-04-25T03:24:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139350K .......... .......... .......... .......... .......... 82% 27.2M 1s 2024-04-25T03:24:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139400K .......... .......... .......... .......... .......... 82% 23.6M 1s 2024-04-25T03:24:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139450K .......... .......... .......... .......... .......... 82% 38.1M 1s 2024-04-25T03:24:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139500K .......... .......... .......... .......... .......... 82% 39.3M 1s 2024-04-25T03:24:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139550K .......... .......... .......... .......... .......... 82% 17.5M 1s 2024-04-25T03:24:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139600K .......... .......... .......... .......... .......... 82% 53.4M 1s 2024-04-25T03:24:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139650K .......... .......... .......... .......... .......... 82% 22.1M 1s 2024-04-25T03:24:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139700K .......... .......... .......... .......... .......... 82% 39.3M 1s 2024-04-25T03:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139750K .......... .......... .......... .......... .......... 82% 19.0M 1s 2024-04-25T03:24:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139800K .......... .......... .......... .......... .......... 82% 55.8M 1s 2024-04-25T03:24:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139850K .......... .......... .......... .......... .......... 82% 28.4M 1s 2024-04-25T03:24:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139900K .......... .......... .......... .......... .......... 83% 21.8M 1s 2024-04-25T03:24:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:139950K .......... .......... .......... .......... .......... 83% 25.8M 1s 2024-04-25T03:24:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140000K .......... .......... .......... .......... .......... 83% 28.8M 1s 2024-04-25T03:24:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140050K .......... .......... .......... .......... .......... 83% 28.4M 1s 2024-04-25T03:24:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140100K .......... .......... .......... .......... .......... 83% 62.8M 1s 2024-04-25T03:24:21.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140150K .......... .......... .......... .......... .......... 83% 25.2M 1s 2024-04-25T03:24:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140200K .......... .......... .......... .......... .......... 83% 19.8M 1s 2024-04-25T03:24:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140250K .......... .......... .......... .......... .......... 83% 43.1M 1s 2024-04-25T03:24:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140300K .......... .......... .......... .......... .......... 83% 25.3M 1s 2024-04-25T03:24:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140350K .......... .......... .......... .......... .......... 83% 23.9M 1s 2024-04-25T03:24:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140400K .......... .......... .......... .......... .......... 83% 50.5M 1s 2024-04-25T03:24:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140450K .......... .......... .......... .......... .......... 83% 22.6M 1s 2024-04-25T03:24:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140500K .......... .......... .......... .......... .......... 83% 24.8M 1s 2024-04-25T03:24:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140550K .......... .......... .......... .......... .......... 83% 35.8M 1s 2024-04-25T03:24:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140600K .......... .......... .......... .......... .......... 83% 46.2M 1s 2024-04-25T03:24:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140650K .......... .......... .......... .......... .......... 83% 27.0M 1s 2024-04-25T03:24:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140700K .......... .......... .......... .......... .......... 83% 36.1M 1s 2024-04-25T03:24:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140750K .......... .......... .......... .......... .......... 83% 23.0M 1s 2024-04-25T03:24:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140800K .......... .......... .......... .......... .......... 83% 32.4M 1s 2024-04-25T03:24:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140850K .......... .......... .......... .......... .......... 83% 19.9M 1s 2024-04-25T03:24:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140900K .......... .......... .......... .......... .......... 83% 36.1M 1s 2024-04-25T03:24:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:140950K .......... .......... .......... .......... .......... 83% 28.2M 1s 2024-04-25T03:24:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141000K .......... .......... .......... .......... .......... 83% 36.9M 1s 2024-04-25T03:24:21.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141050K .......... .......... .......... .......... .......... 83% 24.4M 1s 2024-04-25T03:24:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141100K .......... .......... .......... .......... .......... 83% 48.2M 1s 2024-04-25T03:24:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141150K .......... .......... .......... .......... .......... 83% 23.6M 1s 2024-04-25T03:24:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141200K .......... .......... .......... .......... .......... 83% 24.1M 1s 2024-04-25T03:24:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141250K .......... .......... .......... .......... .......... 83% 38.5M 1s 2024-04-25T03:24:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141300K .......... .......... .......... .......... .......... 83% 35.5M 1s 2024-04-25T03:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141350K .......... .......... .......... .......... .......... 83% 16.9M 1s 2024-04-25T03:24:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141400K .......... .......... .......... .......... .......... 83% 49.6M 1s 2024-04-25T03:24:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141450K .......... .......... .......... .......... .......... 83% 22.6M 1s 2024-04-25T03:24:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141500K .......... .......... .......... .......... .......... 83% 49.9M 1s 2024-04-25T03:24:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141550K .......... .......... .......... .......... .......... 84% 18.2M 1s 2024-04-25T03:24:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141600K .......... .......... .......... .......... .......... 84% 36.3M 1s 2024-04-25T03:24:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141650K .......... .......... .......... .......... .......... 84% 35.4M 1s 2024-04-25T03:24:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141700K .......... .......... .......... .......... .......... 84% 27.5M 1s 2024-04-25T03:24:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141750K .......... .......... .......... .......... .......... 84% 41.8M 1s 2024-04-25T03:24:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141800K .......... .......... .......... .......... .......... 84% 28.3M 1s 2024-04-25T03:24:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141850K .......... .......... .......... .......... .......... 84% 37.2M 1s 2024-04-25T03:24:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141900K .......... .......... .......... .......... .......... 84% 22.4M 1s 2024-04-25T03:24:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:141950K .......... .......... .......... .......... .......... 84% 21.1M 1s 2024-04-25T03:24:21.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142000K .......... .......... .......... .......... .......... 84% 28.2M 1s 2024-04-25T03:24:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142050K .......... .......... .......... .......... .......... 84% 32.3M 1s 2024-04-25T03:24:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142100K .......... .......... .......... .......... .......... 84% 29.0M 1s 2024-04-25T03:24:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142150K .......... .......... .......... .......... .......... 84% 42.5M 1s 2024-04-25T03:24:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142200K .......... .......... .......... .......... .......... 84% 23.5M 1s 2024-04-25T03:24:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142250K .......... .......... .......... .......... .......... 84% 39.4M 1s 2024-04-25T03:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142300K .......... .......... .......... .......... .......... 84% 24.6M 1s 2024-04-25T03:24:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142350K .......... .......... .......... .......... .......... 84% 24.1M 1s 2024-04-25T03:24:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142400K .......... .......... .......... .......... .......... 84% 51.0M 1s 2024-04-25T03:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142450K .......... .......... .......... .......... .......... 84% 22.2M 1s 2024-04-25T03:24:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142500K .......... .......... .......... .......... .......... 84% 32.4M 1s 2024-04-25T03:24:21.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142550K .......... .......... .......... .......... .......... 84% 26.6M 1s 2024-04-25T03:24:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142600K .......... .......... .......... .......... .......... 84% 24.4M 1s 2024-04-25T03:24:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142650K .......... .......... .......... .......... .......... 84% 44.3M 1s 2024-04-25T03:24:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142700K .......... .......... .......... .......... .......... 84% 25.0M 1s 2024-04-25T03:24:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142750K .......... .......... .......... .......... .......... 84% 23.6M 1s 2024-04-25T03:24:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142800K .......... .......... .......... .......... .......... 84% 31.8M 1s 2024-04-25T03:24:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142850K .......... .......... .......... .......... .......... 84% 27.8M 1s 2024-04-25T03:24:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142900K .......... .......... .......... .......... .......... 84% 52.1M 1s 2024-04-25T03:24:21.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:142950K .......... .......... .......... .......... .......... 84% 21.7M 1s 2024-04-25T03:24:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143000K .......... .......... .......... .......... .......... 84% 44.9M 1s 2024-04-25T03:24:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143050K .......... .......... .......... .......... .......... 84% 21.3M 1s 2024-04-25T03:24:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143100K .......... .......... .......... .......... .......... 84% 24.1M 1s 2024-04-25T03:24:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143150K .......... .......... .......... .......... .......... 84% 25.8M 1s 2024-04-25T03:24:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143200K .......... .......... .......... .......... .......... 84% 40.7M 1s 2024-04-25T03:24:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143250K .......... .......... .......... .......... .......... 85% 26.9M 1s 2024-04-25T03:24:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143300K .......... .......... .......... .......... .......... 85% 25.2M 1s 2024-04-25T03:24:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143350K .......... .......... .......... .......... .......... 85% 28.3M 1s 2024-04-25T03:24:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143400K .......... .......... .......... .......... .......... 85% 42.9M 1s 2024-04-25T03:24:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143450K .......... .......... .......... .......... .......... 85% 28.7M 1s 2024-04-25T03:24:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143500K .......... .......... .......... .......... .......... 85% 34.9M 1s 2024-04-25T03:24:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143550K .......... .......... .......... .......... .......... 85% 7.56M 1s 2024-04-25T03:24:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143600K .......... .......... .......... .......... .......... 85% 9.91M 1s 2024-04-25T03:24:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143650K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-04-25T03:24:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143700K .......... .......... .......... .......... .......... 85% 62.7M 1s 2024-04-25T03:24:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143750K .......... .......... .......... .......... .......... 85% 23.7M 1s 2024-04-25T03:24:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143800K .......... .......... .......... .......... .......... 85% 50.7M 1s 2024-04-25T03:24:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143850K .......... .......... .......... .......... .......... 85% 22.5M 1s 2024-04-25T03:24:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143900K .......... .......... .......... .......... .......... 85% 40.6M 1s 2024-04-25T03:24:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:143950K .......... .......... .......... .......... .......... 85% 19.9M 1s 2024-04-25T03:24:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144000K .......... .......... .......... .......... .......... 85% 24.0M 1s 2024-04-25T03:24:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144050K .......... .......... .......... .......... .......... 85% 36.3M 1s 2024-04-25T03:24:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144100K .......... .......... .......... .......... .......... 85% 24.8M 1s 2024-04-25T03:24:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144150K .......... .......... .......... .......... .......... 85% 21.7M 1s 2024-04-25T03:24:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144200K .......... .......... .......... .......... .......... 85% 41.0M 1s 2024-04-25T03:24:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144250K .......... .......... .......... .......... .......... 85% 36.8M 1s 2024-04-25T03:24:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144300K .......... .......... .......... .......... .......... 85% 23.7M 1s 2024-04-25T03:24:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144350K .......... .......... .......... .......... .......... 85% 23.5M 1s 2024-04-25T03:24:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144400K .......... .......... .......... .......... .......... 85% 13.0M 1s 2024-04-25T03:24:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144450K .......... .......... .......... .......... .......... 85% 35.6M 1s 2024-04-25T03:24:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144500K .......... .......... .......... .......... .......... 85% 23.6M 1s 2024-04-25T03:24:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144550K .......... .......... .......... .......... .......... 85% 36.0M 1s 2024-04-25T03:24:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144600K .......... .......... .......... .......... .......... 85% 20.2M 1s 2024-04-25T03:24:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144650K .......... .......... .......... .......... .......... 85% 48.3M 1s 2024-04-25T03:24:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144700K .......... .......... .......... .......... .......... 85% 23.9M 1s 2024-04-25T03:24:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144750K .......... .......... .......... .......... .......... 85% 22.1M 1s 2024-04-25T03:24:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144800K .......... .......... .......... .......... .......... 85% 29.7M 1s 2024-04-25T03:24:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144850K .......... .......... .......... .......... .......... 85% 25.3M 1s 2024-04-25T03:24:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144900K .......... .......... .......... .......... .......... 85% 27.0M 1s 2024-04-25T03:24:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:144950K .......... .......... .......... .......... .......... 86% 36.3M 1s 2024-04-25T03:24:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145000K .......... .......... .......... .......... .......... 86% 24.0M 1s 2024-04-25T03:24:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145050K .......... .......... .......... .......... .......... 86% 34.4M 1s 2024-04-25T03:24:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145100K .......... .......... .......... .......... .......... 86% 30.1M 1s 2024-04-25T03:24:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145150K .......... .......... .......... .......... .......... 86% 24.3M 1s 2024-04-25T03:24:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145200K .......... .......... .......... .......... .......... 86% 43.7M 1s 2024-04-25T03:24:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145250K .......... .......... .......... .......... .......... 86% 22.7M 1s 2024-04-25T03:24:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145300K .......... .......... .......... .......... .......... 86% 25.2M 1s 2024-04-25T03:24:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145350K .......... .......... .......... .......... .......... 86% 36.7M 1s 2024-04-25T03:24:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145400K .......... .......... .......... .......... .......... 86% 27.5M 1s 2024-04-25T03:24:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145450K .......... .......... .......... .......... .......... 86% 37.8M 1s 2024-04-25T03:24:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145500K .......... .......... .......... .......... .......... 86% 25.8M 1s 2024-04-25T03:24:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145550K .......... .......... .......... .......... .......... 86% 21.4M 1s 2024-04-25T03:24:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145600K .......... .......... .......... .......... .......... 86% 41.9M 1s 2024-04-25T03:24:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145650K .......... .......... .......... .......... .......... 86% 24.8M 1s 2024-04-25T03:24:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145700K .......... .......... .......... .......... .......... 86% 39.4M 1s 2024-04-25T03:24:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145750K .......... .......... .......... .......... .......... 86% 28.3M 1s 2024-04-25T03:24:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145800K .......... .......... .......... .......... .......... 86% 24.2M 1s 2024-04-25T03:24:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145850K .......... .......... .......... .......... .......... 86% 32.4M 1s 2024-04-25T03:24:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145900K .......... .......... .......... .......... .......... 86% 22.5M 1s 2024-04-25T03:24:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:145950K .......... .......... .......... .......... .......... 86% 27.6M 1s 2024-04-25T03:24:21.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146000K .......... .......... .......... .......... .......... 86% 43.0M 1s 2024-04-25T03:24:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146050K .......... .......... .......... .......... .......... 86% 25.5M 1s 2024-04-25T03:24:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146100K .......... .......... .......... .......... .......... 86% 38.3M 1s 2024-04-25T03:24:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146150K .......... .......... .......... .......... .......... 86% 23.9M 1s 2024-04-25T03:24:21.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146200K .......... .......... .......... .......... .......... 86% 46.3M 1s 2024-04-25T03:24:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146250K .......... .......... .......... .......... .......... 86% 29.8M 1s 2024-04-25T03:24:21.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146300K .......... .......... .......... .......... .......... 86% 26.7M 1s 2024-04-25T03:24:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146350K .......... .......... .......... .......... .......... 86% 17.7M 1s 2024-04-25T03:24:21.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146400K .......... .......... .......... .......... .......... 86% 37.4M 1s 2024-04-25T03:24:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146450K .......... .......... .......... .......... .......... 86% 24.5M 1s 2024-04-25T03:24:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146500K .......... .......... .......... .......... .......... 86% 36.9M 1s 2024-04-25T03:24:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146550K .......... .......... .......... .......... .......... 86% 25.9M 1s 2024-04-25T03:24:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146600K .......... .......... .......... .......... .......... 86% 41.3M 1s 2024-04-25T03:24:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146650K .......... .......... .......... .......... .......... 87% 29.5M 1s 2024-04-25T03:24:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146700K .......... .......... .......... .......... .......... 87% 35.5M 1s 2024-04-25T03:24:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146750K .......... .......... .......... .......... .......... 87% 24.0M 1s 2024-04-25T03:24:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146800K .......... .......... .......... .......... .......... 87% 28.0M 1s 2024-04-25T03:24:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146850K .......... .......... .......... .......... .......... 87% 26.3M 1s 2024-04-25T03:24:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146900K .......... .......... .......... .......... .......... 87% 28.7M 1s 2024-04-25T03:24:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:146950K .......... .......... .......... .......... .......... 87% 44.6M 1s 2024-04-25T03:24:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147000K .......... .......... .......... .......... .......... 87% 22.6M 1s 2024-04-25T03:24:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147050K .......... .......... .......... .......... .......... 87% 22.9M 1s 2024-04-25T03:24:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147100K .......... .......... .......... .......... .......... 87% 49.9M 1s 2024-04-25T03:24:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147150K .......... .......... .......... .......... .......... 87% 23.1M 1s 2024-04-25T03:24:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147200K .......... .......... .......... .......... .......... 87% 24.5M 1s 2024-04-25T03:24:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147250K .......... .......... .......... .......... .......... 87% 42.1M 1s 2024-04-25T03:24:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147300K .......... .......... .......... .......... .......... 87% 23.3M 1s 2024-04-25T03:24:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147350K .......... .......... .......... .......... .......... 87% 44.0M 1s 2024-04-25T03:24:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147400K .......... .......... .......... .......... .......... 87% 21.2M 1s 2024-04-25T03:24:21.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147450K .......... .......... .......... .......... .......... 87% 54.9M 1s 2024-04-25T03:24:21.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147500K .......... .......... .......... .......... .......... 87% 22.8M 1s 2024-04-25T03:24:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147550K .......... .......... .......... .......... .......... 87% 23.8M 1s 2024-04-25T03:24:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147600K .......... .......... .......... .......... .......... 87% 34.7M 1s 2024-04-25T03:24:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147650K .......... .......... .......... .......... .......... 87% 26.8M 1s 2024-04-25T03:24:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147700K .......... .......... .......... .......... .......... 87% 21.2M 1s 2024-04-25T03:24:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147750K .......... .......... .......... .......... .......... 87% 49.2M 1s 2024-04-25T03:24:21.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147800K .......... .......... .......... .......... .......... 87% 24.6M 1s 2024-04-25T03:24:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147850K .......... .......... .......... .......... .......... 87% 38.3M 1s 2024-04-25T03:24:21.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147900K .......... .......... .......... .......... .......... 87% 23.2M 1s 2024-04-25T03:24:21.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:147950K .......... .......... .......... .......... .......... 87% 29.8M 1s 2024-04-25T03:24:21.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148000K .......... .......... .......... .......... .......... 87% 35.5M 1s 2024-04-25T03:24:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148050K .......... .......... .......... .......... .......... 87% 24.9M 1s 2024-04-25T03:24:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148100K .......... .......... .......... .......... .......... 87% 37.8M 1s 2024-04-25T03:24:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148150K .......... .......... .......... .......... .......... 87% 23.7M 1s 2024-04-25T03:24:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148200K .......... .......... .......... .......... .......... 87% 22.1M 1s 2024-04-25T03:24:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148250K .......... .......... .......... .......... .......... 87% 46.8M 1s 2024-04-25T03:24:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148300K .......... .......... .......... .......... .......... 88% 34.6M 1s 2024-04-25T03:24:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148350K .......... .......... .......... .......... .......... 88% 23.0M 1s 2024-04-25T03:24:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148400K .......... .......... .......... .......... .......... 88% 29.3M 1s 2024-04-25T03:24:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148450K .......... .......... .......... .......... .......... 88% 26.7M 1s 2024-04-25T03:24:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148500K .......... .......... .......... .......... .......... 88% 46.5M 1s 2024-04-25T03:24:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148550K .......... .......... .......... .......... .......... 88% 23.1M 1s 2024-04-25T03:24:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148600K .......... .......... .......... .......... .......... 88% 37.2M 1s 2024-04-25T03:24:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148650K .......... .......... .......... .......... .......... 88% 25.0M 1s 2024-04-25T03:24:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148700K .......... .......... .......... .......... .......... 88% 21.7M 1s 2024-04-25T03:24:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148750K .......... .......... .......... .......... .......... 88% 24.8M 1s 2024-04-25T03:24:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148800K .......... .......... .......... .......... .......... 88% 43.5M 1s 2024-04-25T03:24:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148850K .......... .......... .......... .......... .......... 88% 32.9M 1s 2024-04-25T03:24:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148900K .......... .......... .......... .......... .......... 88% 24.0M 1s 2024-04-25T03:24:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:148950K .......... .......... .......... .......... .......... 88% 21.9M 1s 2024-04-25T03:24:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149000K .......... .......... .......... .......... .......... 88% 58.4M 1s 2024-04-25T03:24:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149050K .......... .......... .......... .......... .......... 88% 33.4M 1s 2024-04-25T03:24:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149100K .......... .......... .......... .......... .......... 88% 25.3M 1s 2024-04-25T03:24:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149150K .......... .......... .......... .......... .......... 88% 10.7M 1s 2024-04-25T03:24:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149200K .......... .......... .......... .......... .......... 88% 28.4M 1s 2024-04-25T03:24:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149250K .......... .......... .......... .......... .......... 88% 23.3M 1s 2024-04-25T03:24:21.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149300K .......... .......... .......... .......... .......... 88% 49.0M 1s 2024-04-25T03:24:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149350K .......... .......... .......... .......... .......... 88% 21.8M 1s 2024-04-25T03:24:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149400K .......... .......... .......... .......... .......... 88% 20.2M 1s 2024-04-25T03:24:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149450K .......... .......... .......... .......... .......... 88% 39.3M 1s 2024-04-25T03:24:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149500K .......... .......... .......... .......... .......... 88% 30.2M 1s 2024-04-25T03:24:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149550K .......... .......... .......... .......... .......... 88% 26.9M 1s 2024-04-25T03:24:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149600K .......... .......... .......... .......... .......... 88% 30.3M 1s 2024-04-25T03:24:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149650K .......... .......... .......... .......... .......... 88% 21.3M 1s 2024-04-25T03:24:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149700K .......... .......... .......... .......... .......... 88% 32.7M 1s 2024-04-25T03:24:21.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149750K .......... .......... .......... .......... .......... 88% 45.4M 1s 2024-04-25T03:24:21.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149800K .......... .......... .......... .......... .......... 88% 24.3M 1s 2024-04-25T03:24:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149850K .......... .......... .......... .......... .......... 88% 29.5M 1s 2024-04-25T03:24:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149900K .......... .......... .......... .......... .......... 88% 28.6M 1s 2024-04-25T03:24:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:149950K .......... .......... .......... .......... .......... 88% 25.9M 1s 2024-04-25T03:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150000K .......... .......... .......... .......... .......... 89% 22.1M 1s 2024-04-25T03:24:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150050K .......... .......... .......... .......... .......... 89% 46.1M 1s 2024-04-25T03:24:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150100K .......... .......... .......... .......... .......... 89% 23.7M 1s 2024-04-25T03:24:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150150K .......... .......... .......... .......... .......... 89% 33.3M 1s 2024-04-25T03:24:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150200K .......... .......... .......... .......... .......... 89% 30.1M 1s 2024-04-25T03:24:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150250K .......... .......... .......... .......... .......... 89% 42.0M 1s 2024-04-25T03:24:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150300K .......... .......... .......... .......... .......... 89% 29.7M 1s 2024-04-25T03:24:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150350K .......... .......... .......... .......... .......... 89% 18.6M 1s 2024-04-25T03:24:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150400K .......... .......... .......... .......... .......... 89% 36.0M 1s 2024-04-25T03:24:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150450K .......... .......... .......... .......... .......... 89% 24.0M 1s 2024-04-25T03:24:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150500K .......... .......... .......... .......... .......... 89% 24.3M 1s 2024-04-25T03:24:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150550K .......... .......... .......... .......... .......... 89% 46.2M 1s 2024-04-25T03:24:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150600K .......... .......... .......... .......... .......... 89% 23.6M 1s 2024-04-25T03:24:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150650K .......... .......... .......... .......... .......... 89% 38.1M 1s 2024-04-25T03:24:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150700K .......... .......... .......... .......... .......... 89% 25.1M 1s 2024-04-25T03:24:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150750K .......... .......... .......... .......... .......... 89% 22.4M 1s 2024-04-25T03:24:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150800K .......... .......... .......... .......... .......... 89% 54.1M 1s 2024-04-25T03:24:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150850K .......... .......... .......... .......... .......... 89% 24.0M 1s 2024-04-25T03:24:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150900K .......... .......... .......... .......... .......... 89% 39.1M 1s 2024-04-25T03:24:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:150950K .......... .......... .......... .......... .......... 89% 24.6M 1s 2024-04-25T03:24:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151000K .......... .......... .......... .......... .......... 89% 23.2M 1s 2024-04-25T03:24:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151050K .......... .......... .......... .......... .......... 89% 35.5M 1s 2024-04-25T03:24:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151100K .......... .......... .......... .......... .......... 89% 27.4M 1s 2024-04-25T03:24:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151150K .......... .......... .......... .......... .......... 89% 25.8M 1s 2024-04-25T03:24:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151200K .......... .......... .......... .......... .......... 89% 40.9M 1s 2024-04-25T03:24:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151250K .......... .......... .......... .......... .......... 89% 24.2M 1s 2024-04-25T03:24:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151300K .......... .......... .......... .......... .......... 89% 37.1M 1s 2024-04-25T03:24:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151350K .......... .......... .......... .......... .......... 89% 27.5M 1s 2024-04-25T03:24:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151400K .......... .......... .......... .......... .......... 89% 31.6M 1s 2024-04-25T03:24:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151450K .......... .......... .......... .......... .......... 89% 28.7M 1s 2024-04-25T03:24:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151500K .......... .......... .......... .......... .......... 89% 41.8M 1s 2024-04-25T03:24:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151550K .......... .......... .......... .......... .......... 89% 16.1M 1s 2024-04-25T03:24:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151600K .......... .......... .......... .......... .......... 89% 47.5M 1s 2024-04-25T03:24:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151650K .......... .......... .......... .......... .......... 89% 24.5M 1s 2024-04-25T03:24:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151700K .......... .......... .......... .......... .......... 90% 39.2M 1s 2024-04-25T03:24:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151750K .......... .......... .......... .......... .......... 90% 24.7M 1s 2024-04-25T03:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151800K .......... .......... .......... .......... .......... 90% 24.1M 1s 2024-04-25T03:24:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151850K .......... .......... .......... .......... .......... 90% 42.1M 1s 2024-04-25T03:24:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151900K .......... .......... .......... .......... .......... 90% 41.4M 1s 2024-04-25T03:24:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:151950K .......... .......... .......... .......... .......... 90% 23.5M 1s 2024-04-25T03:24:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152000K .......... .......... .......... .......... .......... 90% 25.1M 1s 2024-04-25T03:24:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152050K .......... .......... .......... .......... .......... 90% 39.0M 1s 2024-04-25T03:24:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152100K .......... .......... .......... .......... .......... 90% 25.9M 1s 2024-04-25T03:24:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152150K .......... .......... .......... .......... .......... 90% 23.2M 1s 2024-04-25T03:24:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152200K .......... .......... .......... .......... .......... 90% 40.5M 1s 2024-04-25T03:24:21.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152250K .......... .......... .......... .......... .......... 90% 24.1M 1s 2024-04-25T03:24:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152300K .......... .......... .......... .......... .......... 90% 37.8M 1s 2024-04-25T03:24:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152350K .......... .......... .......... .......... .......... 90% 26.0M 1s 2024-04-25T03:24:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152400K .......... .......... .......... .......... .......... 90% 27.8M 1s 2024-04-25T03:24:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152450K .......... .......... .......... .......... .......... 90% 32.8M 1s 2024-04-25T03:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152500K .......... .......... .......... .......... .......... 90% 24.0M 1s 2024-04-25T03:24:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152550K .......... .......... .......... .......... .......... 90% 52.4M 1s 2024-04-25T03:24:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152600K .......... .......... .......... .......... .......... 90% 25.2M 1s 2024-04-25T03:24:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152650K .......... .......... .......... .......... .......... 90% 29.9M 1s 2024-04-25T03:24:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152700K .......... .......... .......... .......... .......... 90% 23.6M 1s 2024-04-25T03:24:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152750K .......... .......... .......... .......... .......... 90% 21.8M 1s 2024-04-25T03:24:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152800K .......... .......... .......... .......... .......... 90% 25.6M 1s 2024-04-25T03:24:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152850K .......... .......... .......... .......... .......... 90% 41.5M 1s 2024-04-25T03:24:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152900K .......... .......... .......... .......... .......... 90% 25.5M 1s 2024-04-25T03:24:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:152950K .......... .......... .......... .......... .......... 90% 38.0M 1s 2024-04-25T03:24:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153000K .......... .......... .......... .......... .......... 90% 22.4M 1s 2024-04-25T03:24:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153050K .......... .......... .......... .......... .......... 90% 55.4M 1s 2024-04-25T03:24:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153100K .......... .......... .......... .......... .......... 90% 23.0M 1s 2024-04-25T03:24:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153150K .......... .......... .......... .......... .......... 90% 23.1M 1s 2024-04-25T03:24:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153200K .......... .......... .......... .......... .......... 90% 36.5M 1s 2024-04-25T03:24:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153250K .......... .......... .......... .......... .......... 90% 25.3M 1s 2024-04-25T03:24:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153300K .......... .......... .......... .......... .......... 90% 23.8M 1s 2024-04-25T03:24:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153350K .......... .......... .......... .......... .......... 91% 40.8M 1s 2024-04-25T03:24:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153400K .......... .......... .......... .......... .......... 91% 28.0M 1s 2024-04-25T03:24:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153450K .......... .......... .......... .......... .......... 91% 37.3M 1s 2024-04-25T03:24:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153500K .......... .......... .......... .......... .......... 91% 20.8M 1s 2024-04-25T03:24:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153550K .......... .......... .......... .......... .......... 91% 23.2M 1s 2024-04-25T03:24:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153600K .......... .......... .......... .......... .......... 91% 52.6M 1s 2024-04-25T03:24:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153650K .......... .......... .......... .......... .......... 91% 24.3M 1s 2024-04-25T03:24:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153700K .......... .......... .......... .......... .......... 91% 31.7M 1s 2024-04-25T03:24:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153750K .......... .......... .......... .......... .......... 91% 28.2M 1s 2024-04-25T03:24:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153800K .......... .......... .......... .......... .......... 91% 23.4M 1s 2024-04-25T03:24:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153850K .......... .......... .......... .......... .......... 91% 48.6M 1s 2024-04-25T03:24:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153900K .......... .......... .......... .......... .......... 91% 38.2M 1s 2024-04-25T03:24:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:153950K .......... .......... .......... .......... .......... 91% 17.3M 1s 2024-04-25T03:24:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154000K .......... .......... .......... .......... .......... 91% 36.0M 1s 2024-04-25T03:24:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154050K .......... .......... .......... .......... .......... 91% 23.0M 1s 2024-04-25T03:24:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154100K .......... .......... .......... .......... .......... 91% 59.9M 1s 2024-04-25T03:24:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154150K .......... .......... .......... .......... .......... 91% 17.6M 1s 2024-04-25T03:24:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154200K .......... .......... .......... .......... .......... 91% 51.2M 1s 2024-04-25T03:24:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154250K .......... .......... .......... .......... .......... 91% 22.7M 1s 2024-04-25T03:24:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154300K .......... .......... .......... .......... .......... 91% 44.3M 1s 2024-04-25T03:24:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154350K .......... .......... .......... .......... .......... 91% 6.49M 1s 2024-04-25T03:24:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154400K .......... .......... .......... .......... .......... 91% 50.0M 1s 2024-04-25T03:24:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154450K .......... .......... .......... .......... .......... 91% 23.7M 1s 2024-04-25T03:24:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154500K .......... .......... .......... .......... .......... 91% 39.4M 1s 2024-04-25T03:24:21.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154550K .......... .......... .......... .......... .......... 91% 23.3M 1s 2024-04-25T03:24:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154600K .......... .......... .......... .......... .......... 91% 42.4M 0s 2024-04-25T03:24:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154650K .......... .......... .......... .......... .......... 91% 26.5M 0s 2024-04-25T03:24:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154700K .......... .......... .......... .......... .......... 91% 40.7M 0s 2024-04-25T03:24:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154750K .......... .......... .......... .......... .......... 91% 20.3M 0s 2024-04-25T03:24:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154800K .......... .......... .......... .......... .......... 91% 27.6M 0s 2024-04-25T03:24:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154850K .......... .......... .......... .......... .......... 91% 49.6M 0s 2024-04-25T03:24:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154900K .......... .......... .......... .......... .......... 91% 24.9M 0s 2024-04-25T03:24:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:154950K .......... .......... .......... .......... .......... 91% 34.0M 0s 2024-04-25T03:24:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155000K .......... .......... .......... .......... .......... 91% 24.4M 0s 2024-04-25T03:24:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155050K .......... .......... .......... .......... .......... 92% 23.7M 0s 2024-04-25T03:24:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155100K .......... .......... .......... .......... .......... 92% 44.4M 0s 2024-04-25T03:24:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155150K .......... .......... .......... .......... .......... 92% 24.9M 0s 2024-04-25T03:24:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155200K .......... .......... .......... .......... .......... 92% 24.5M 0s 2024-04-25T03:24:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155250K .......... .......... .......... .......... .......... 92% 37.6M 0s 2024-04-25T03:24:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155300K .......... .......... .......... .......... .......... 92% 24.0M 0s 2024-04-25T03:24:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155350K .......... .......... .......... .......... .......... 92% 37.9M 0s 2024-04-25T03:24:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155400K .......... .......... .......... .......... .......... 92% 28.2M 0s 2024-04-25T03:24:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155450K .......... .......... .......... .......... .......... 92% 24.2M 0s 2024-04-25T03:24:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155500K .......... .......... .......... .......... .......... 92% 64.6M 0s 2024-04-25T03:24:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155550K .......... .......... .......... .......... .......... 92% 18.7M 0s 2024-04-25T03:24:22.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155600K .......... .......... .......... .......... .......... 92% 24.5M 0s 2024-04-25T03:24:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155650K .......... .......... .......... .......... .......... 92% 39.6M 0s 2024-04-25T03:24:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155700K .......... .......... .......... .......... .......... 92% 26.8M 0s 2024-04-25T03:24:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155750K .......... .......... .......... .......... .......... 92% 41.2M 0s 2024-04-25T03:24:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155800K .......... .......... .......... .......... .......... 92% 28.7M 0s 2024-04-25T03:24:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155850K .......... .......... .......... .......... .......... 92% 34.7M 0s 2024-04-25T03:24:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155900K .......... .......... .......... .......... .......... 92% 23.3M 0s 2024-04-25T03:24:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:155950K .......... .......... .......... .......... .......... 92% 27.9M 0s 2024-04-25T03:24:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156000K .......... .......... .......... .......... .......... 92% 22.2M 0s 2024-04-25T03:24:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156050K .......... .......... .......... .......... .......... 92% 37.6M 0s 2024-04-25T03:24:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156100K .......... .......... .......... .......... .......... 92% 28.2M 0s 2024-04-25T03:24:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156150K .......... .......... .......... .......... .......... 92% 43.4M 0s 2024-04-25T03:24:22.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156200K .......... .......... .......... .......... .......... 92% 19.5M 0s 2024-04-25T03:24:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156250K .......... .......... .......... .......... .......... 92% 48.1M 0s 2024-04-25T03:24:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156300K .......... .......... .......... .......... .......... 92% 22.5M 0s 2024-04-25T03:24:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156350K .......... .......... .......... .......... .......... 92% 25.4M 0s 2024-04-25T03:24:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156400K .......... .......... .......... .......... .......... 92% 35.6M 0s 2024-04-25T03:24:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156450K .......... .......... .......... .......... .......... 92% 24.5M 0s 2024-04-25T03:24:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156500K .......... .......... .......... .......... .......... 92% 42.9M 0s 2024-04-25T03:24:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156550K .......... .......... .......... .......... .......... 92% 27.8M 0s 2024-04-25T03:24:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156600K .......... .......... .......... .......... .......... 92% 31.1M 0s 2024-04-25T03:24:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156650K .......... .......... .......... .......... .......... 92% 9.48M 0s 2024-04-25T03:24:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156700K .......... .......... .......... .......... .......... 92% 35.8M 0s 2024-04-25T03:24:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156750K .......... .......... .......... .......... .......... 93% 17.1M 0s 2024-04-25T03:24:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156800K .......... .......... .......... .......... .......... 93% 29.0M 0s 2024-04-25T03:24:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156850K .......... .......... .......... .......... .......... 93% 13.3M 0s 2024-04-25T03:24:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156900K .......... .......... .......... .......... .......... 93% 42.4M 0s 2024-04-25T03:24:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:156950K .......... .......... .......... .......... .......... 93% 24.6M 0s 2024-04-25T03:24:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157000K .......... .......... .......... .......... .......... 93% 40.6M 0s 2024-04-25T03:24:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157050K .......... .......... .......... .......... .......... 93% 21.4M 0s 2024-04-25T03:24:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157100K .......... .......... .......... .......... .......... 93% 56.9M 0s 2024-04-25T03:24:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157150K .......... .......... .......... .......... .......... 93% 21.5M 0s 2024-04-25T03:24:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157200K .......... .......... .......... .......... .......... 93% 26.5M 0s 2024-04-25T03:24:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157250K .......... .......... .......... .......... .......... 93% 24.3M 0s 2024-04-25T03:24:22.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157300K .......... .......... .......... .......... .......... 93% 40.6M 0s 2024-04-25T03:24:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157350K .......... .......... .......... .......... .......... 93% 34.9M 0s 2024-04-25T03:24:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157400K .......... .......... .......... .......... .......... 93% 30.8M 0s 2024-04-25T03:24:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157450K .......... .......... .......... .......... .......... 93% 23.3M 0s 2024-04-25T03:24:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157500K .......... .......... .......... .......... .......... 93% 39.2M 0s 2024-04-25T03:24:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157550K .......... .......... .......... .......... .......... 93% 18.7M 0s 2024-04-25T03:24:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157600K .......... .......... .......... .......... .......... 93% 34.3M 0s 2024-04-25T03:24:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157650K .......... .......... .......... .......... .......... 93% 47.3M 0s 2024-04-25T03:24:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157700K .......... .......... .......... .......... .......... 93% 24.1M 0s 2024-04-25T03:24:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157750K .......... .......... .......... .......... .......... 93% 35.3M 0s 2024-04-25T03:24:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157800K .......... .......... .......... .......... .......... 93% 24.5M 0s 2024-04-25T03:24:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157850K .......... .......... .......... .......... .......... 93% 22.7M 0s 2024-04-25T03:24:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157900K .......... .......... .......... .......... .......... 93% 55.0M 0s 2024-04-25T03:24:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:157950K .......... .......... .......... .......... .......... 93% 26.4M 0s 2024-04-25T03:24:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158000K .......... .......... .......... .......... .......... 93% 22.3M 0s 2024-04-25T03:24:22.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158050K .......... .......... .......... .......... .......... 93% 31.6M 0s 2024-04-25T03:24:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158100K .......... .......... .......... .......... .......... 93% 28.9M 0s 2024-04-25T03:24:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158150K .......... .......... .......... .......... .......... 93% 36.4M 0s 2024-04-25T03:24:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158200K .......... .......... .......... .......... .......... 93% 35.4M 0s 2024-04-25T03:24:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158250K .......... .......... .......... .......... .......... 93% 17.7M 0s 2024-04-25T03:24:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158300K .......... .......... .......... .......... .......... 93% 114M 0s 2024-04-25T03:24:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158350K .......... .......... .......... .......... .......... 93% 19.1M 0s 2024-04-25T03:24:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158400K .......... .......... .......... .......... .......... 93% 23.0M 0s 2024-04-25T03:24:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158450K .......... .......... .......... .......... .......... 94% 43.1M 0s 2024-04-25T03:24:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158500K .......... .......... .......... .......... .......... 94% 26.7M 0s 2024-04-25T03:24:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158550K .......... .......... .......... .......... .......... 94% 34.4M 0s 2024-04-25T03:24:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158600K .......... .......... .......... .......... .......... 94% 26.3M 0s 2024-04-25T03:24:22.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158650K .......... .......... .......... .......... .......... 94% 42.5M 0s 2024-04-25T03:24:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158700K .......... .......... .......... .......... .......... 94% 35.3M 0s 2024-04-25T03:24:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158750K .......... .......... .......... .......... .......... 94% 20.4M 0s 2024-04-25T03:24:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158800K .......... .......... .......... .......... .......... 94% 35.3M 0s 2024-04-25T03:24:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158850K .......... .......... .......... .......... .......... 94% 24.8M 0s 2024-04-25T03:24:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158900K .......... .......... .......... .......... .......... 94% 22.6M 0s 2024-04-25T03:24:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:158950K .......... .......... .......... .......... .......... 94% 45.2M 0s 2024-04-25T03:24:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159000K .......... .......... .......... .......... .......... 94% 24.7M 0s 2024-04-25T03:24:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159050K .......... .......... .......... .......... .......... 94% 39.8M 0s 2024-04-25T03:24:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159100K .......... .......... .......... .......... .......... 94% 24.2M 0s 2024-04-25T03:24:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159150K .......... .......... .......... .......... .......... 94% 24.0M 0s 2024-04-25T03:24:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159200K .......... .......... .......... .......... .......... 94% 46.0M 0s 2024-04-25T03:24:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159250K .......... .......... .......... .......... .......... 94% 26.5M 0s 2024-04-25T03:24:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159300K .......... .......... .......... .......... .......... 94% 32.7M 0s 2024-04-25T03:24:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159350K .......... .......... .......... .......... .......... 94% 25.8M 0s 2024-04-25T03:24:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159400K .......... .......... .......... .......... .......... 94% 38.5M 0s 2024-04-25T03:24:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159450K .......... .......... .......... .......... .......... 94% 23.8M 0s 2024-04-25T03:24:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159500K .......... .......... .......... .......... .......... 94% 20.4M 0s 2024-04-25T03:24:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159550K .......... .......... .......... .......... .......... 94% 20.1M 0s 2024-04-25T03:24:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159600K .......... .......... .......... .......... .......... 94% 32.8M 0s 2024-04-25T03:24:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159650K .......... .......... .......... .......... .......... 94% 28.0M 0s 2024-04-25T03:24:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159700K .......... .......... .......... .......... .......... 94% 33.6M 0s 2024-04-25T03:24:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159750K .......... .......... .......... .......... .......... 94% 10.2M 0s 2024-04-25T03:24:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159800K .......... .......... .......... .......... .......... 94% 45.8M 0s 2024-04-25T03:24:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159850K .......... .......... .......... .......... .......... 94% 23.9M 0s 2024-04-25T03:24:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159900K .......... .......... .......... .......... .......... 94% 45.8M 0s 2024-04-25T03:24:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:159950K .......... .......... .......... .......... .......... 94% 23.7M 0s 2024-04-25T03:24:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160000K .......... .......... .......... .......... .......... 94% 23.8M 0s 2024-04-25T03:24:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160050K .......... .......... .......... .......... .......... 94% 39.2M 0s 2024-04-25T03:24:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160100K .......... .......... .......... .......... .......... 95% 22.2M 0s 2024-04-25T03:24:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160150K .......... .......... .......... .......... .......... 95% 22.9M 0s 2024-04-25T03:24:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160200K .......... .......... .......... .......... .......... 95% 58.0M 0s 2024-04-25T03:24:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160250K .......... .......... .......... .......... .......... 95% 22.6M 0s 2024-04-25T03:24:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160300K .......... .......... .......... .......... .......... 95% 39.7M 0s 2024-04-25T03:24:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160350K .......... .......... .......... .......... .......... 95% 23.2M 0s 2024-04-25T03:24:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160400K .......... .......... .......... .......... .......... 95% 26.4M 0s 2024-04-25T03:24:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160450K .......... .......... .......... .......... .......... 95% 38.5M 0s 2024-04-25T03:24:22.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160500K .......... .......... .......... .......... .......... 95% 25.0M 0s 2024-04-25T03:24:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160550K .......... .......... .......... .......... .......... 95% 33.1M 0s 2024-04-25T03:24:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160600K .......... .......... .......... .......... .......... 95% 27.4M 0s 2024-04-25T03:24:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160650K .......... .......... .......... .......... .......... 95% 22.5M 0s 2024-04-25T03:24:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160700K .......... .......... .......... .......... .......... 95% 53.4M 0s 2024-04-25T03:24:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160750K .......... .......... .......... .......... .......... 95% 24.1M 0s 2024-04-25T03:24:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160800K .......... .......... .......... .......... .......... 95% 24.5M 0s 2024-04-25T03:24:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160850K .......... .......... .......... .......... .......... 95% 36.7M 0s 2024-04-25T03:24:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160900K .......... .......... .......... .......... .......... 95% 23.4M 0s 2024-04-25T03:24:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:160950K .......... .......... .......... .......... .......... 95% 44.7M 0s 2024-04-25T03:24:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161000K .......... .......... .......... .......... .......... 95% 24.5M 0s 2024-04-25T03:24:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161050K .......... .......... .......... .......... .......... 95% 30.8M 0s 2024-04-25T03:24:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161100K .......... .......... .......... .......... .......... 95% 30.0M 0s 2024-04-25T03:24:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161150K .......... .......... .......... .......... .......... 95% 21.8M 0s 2024-04-25T03:24:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161200K .......... .......... .......... .......... .......... 95% 23.1M 0s 2024-04-25T03:24:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161250K .......... .......... .......... .......... .......... 95% 45.7M 0s 2024-04-25T03:24:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161300K .......... .......... .......... .......... .......... 95% 23.4M 0s 2024-04-25T03:24:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161350K .......... .......... .......... .......... .......... 95% 42.5M 0s 2024-04-25T03:24:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161400K .......... .......... .......... .......... .......... 95% 28.4M 0s 2024-04-25T03:24:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161450K .......... .......... .......... .......... .......... 95% 33.5M 0s 2024-04-25T03:24:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161500K .......... .......... .......... .......... .......... 95% 25.6M 0s 2024-04-25T03:24:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161550K .......... .......... .......... .......... .......... 95% 22.7M 0s 2024-04-25T03:24:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161600K .......... .......... .......... .......... .......... 95% 40.2M 0s 2024-04-25T03:24:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161650K .......... .......... .......... .......... .......... 95% 24.4M 0s 2024-04-25T03:24:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161700K .......... .......... .......... .......... .......... 95% 35.5M 0s 2024-04-25T03:24:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161750K .......... .......... .......... .......... .......... 95% 26.9M 0s 2024-04-25T03:24:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161800K .......... .......... .......... .......... .......... 96% 23.5M 0s 2024-04-25T03:24:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161850K .......... .......... .......... .......... .......... 96% 40.6M 0s 2024-04-25T03:24:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161900K .......... .......... .......... .......... .......... 96% 24.6M 0s 2024-04-25T03:24:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:161950K .......... .......... .......... .......... .......... 96% 28.1M 0s 2024-04-25T03:24:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162000K .......... .......... .......... .......... .......... 96% 31.5M 0s 2024-04-25T03:24:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162050K .......... .......... .......... .......... .......... 96% 25.3M 0s 2024-04-25T03:24:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162100K .......... .......... .......... .......... .......... 96% 40.3M 0s 2024-04-25T03:24:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162150K .......... .......... .......... .......... .......... 96% 23.0M 0s 2024-04-25T03:24:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162200K .......... .......... .......... .......... .......... 96% 41.5M 0s 2024-04-25T03:24:22.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162250K .......... .......... .......... .......... .......... 96% 23.5M 0s 2024-04-25T03:24:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162300K .......... .......... .......... .......... .......... 96% 41.5M 0s 2024-04-25T03:24:22.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162350K .......... .......... .......... .......... .......... 96% 18.7M 0s 2024-04-25T03:24:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162400K .......... .......... .......... .......... .......... 96% 43.8M 0s 2024-04-25T03:24:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162450K .......... .......... .......... .......... .......... 96% 22.1M 0s 2024-04-25T03:24:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162500K .......... .......... .......... .......... .......... 96% 46.6M 0s 2024-04-25T03:24:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162550K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-04-25T03:24:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162600K .......... .......... .......... .......... .......... 96% 35.8M 0s 2024-04-25T03:24:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162650K .......... .......... .......... .......... .......... 96% 27.5M 0s 2024-04-25T03:24:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162700K .......... .......... .......... .......... .......... 96% 45.0M 0s 2024-04-25T03:24:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162750K .......... .......... .......... .......... .......... 96% 23.5M 0s 2024-04-25T03:24:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162800K .......... .......... .......... .......... .......... 96% 23.4M 0s 2024-04-25T03:24:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162850K .......... .......... .......... .......... .......... 96% 40.7M 0s 2024-04-25T03:24:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162900K .......... .......... .......... .......... .......... 96% 23.1M 0s 2024-04-25T03:24:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:162950K .......... .......... .......... .......... .......... 96% 23.2M 0s 2024-04-25T03:24:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163000K .......... .......... .......... .......... .......... 96% 42.1M 0s 2024-04-25T03:24:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163050K .......... .......... .......... .......... .......... 96% 30.1M 0s 2024-04-25T03:24:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163100K .......... .......... .......... .......... .......... 96% 34.0M 0s 2024-04-25T03:24:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163150K .......... .......... .......... .......... .......... 96% 24.1M 0s 2024-04-25T03:24:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163200K .......... .......... .......... .......... .......... 96% 24.1M 0s 2024-04-25T03:24:22.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163250K .......... .......... .......... .......... .......... 96% 46.5M 0s 2024-04-25T03:24:22.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163300K .......... .......... .......... .......... .......... 96% 23.4M 0s 2024-04-25T03:24:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163350K .......... .......... .......... .......... .......... 96% 40.8M 0s 2024-04-25T03:24:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163400K .......... .......... .......... .......... .......... 96% 22.3M 0s 2024-04-25T03:24:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163450K .......... .......... .......... .......... .......... 96% 22.3M 0s 2024-04-25T03:24:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163500K .......... .......... .......... .......... .......... 97% 53.7M 0s 2024-04-25T03:24:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163550K .......... .......... .......... .......... .......... 97% 23.2M 0s 2024-04-25T03:24:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163600K .......... .......... .......... .......... .......... 97% 25.0M 0s 2024-04-25T03:24:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163650K .......... .......... .......... .......... .......... 97% 38.1M 0s 2024-04-25T03:24:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163700K .......... .......... .......... .......... .......... 97% 24.3M 0s 2024-04-25T03:24:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163750K .......... .......... .......... .......... .......... 97% 37.9M 0s 2024-04-25T03:24:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163800K .......... .......... .......... .......... .......... 97% 24.8M 0s 2024-04-25T03:24:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163850K .......... .......... .......... .......... .......... 97% 43.1M 0s 2024-04-25T03:24:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163900K .......... .......... .......... .......... .......... 97% 23.1M 0s 2024-04-25T03:24:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:163950K .......... .......... .......... .......... .......... 97% 26.0M 0s 2024-04-25T03:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164000K .......... .......... .......... .......... .......... 97% 22.9M 0s 2024-04-25T03:24:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164050K .......... .......... .......... .......... .......... 97% 53.8M 0s 2024-04-25T03:24:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164100K .......... .......... .......... .......... .......... 97% 23.1M 0s 2024-04-25T03:24:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164150K .......... .......... .......... .......... .......... 97% 38.9M 0s 2024-04-25T03:24:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164200K .......... .......... .......... .......... .......... 97% 21.5M 0s 2024-04-25T03:24:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164250K .......... .......... .......... .......... .......... 97% 39.3M 0s 2024-04-25T03:24:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164300K .......... .......... .......... .......... .......... 97% 30.4M 0s 2024-04-25T03:24:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164350K .......... .......... .......... .......... .......... 97% 23.5M 0s 2024-04-25T03:24:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164400K .......... .......... .......... .......... .......... 97% 36.0M 0s 2024-04-25T03:24:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164450K .......... .......... .......... .......... .......... 97% 24.5M 0s 2024-04-25T03:24:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164500K .......... .......... .......... .......... .......... 97% 49.9M 0s 2024-04-25T03:24:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164550K .......... .......... .......... .......... .......... 97% 21.2M 0s 2024-04-25T03:24:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164600K .......... .......... .......... .......... .......... 97% 43.0M 0s 2024-04-25T03:24:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164650K .......... .......... .......... .......... .......... 97% 21.2M 0s 2024-04-25T03:24:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164700K .......... .......... .......... .......... .......... 97% 27.3M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164750K .......... .......... .......... .......... .......... 97% 22.5M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164800K .......... .......... .......... .......... .......... 97% 54.0M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164850K .......... .......... .......... .......... .......... 97% 24.1M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164900K .......... .......... .......... .......... .......... 97% 33.2M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:164950K .......... .......... .......... .......... .......... 97% 26.0M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165000K .......... .......... .......... .......... .......... 97% 54.4M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165050K .......... .......... .......... .......... .......... 97% 21.1M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165100K .......... .......... .......... .......... .......... 97% 39.5M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165150K .......... .......... .......... .......... .......... 98% 22.3M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165200K .......... .......... .......... .......... .......... 98% 27.5M 0s 2024-04-25T03:24:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165250K .......... .......... .......... .......... .......... 98% 27.4M 0s 2024-04-25T03:24:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165300K .......... .......... .......... .......... .......... 98% 31.3M 0s 2024-04-25T03:24:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165350K .......... .......... .......... .......... .......... 98% 28.1M 0s 2024-04-25T03:24:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165400K .......... .......... .......... .......... .......... 98% 54.7M 0s 2024-04-25T03:24:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165450K .......... .......... .......... .......... .......... 98% 19.3M 0s 2024-04-25T03:24:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165500K .......... .......... .......... .......... .......... 98% 59.4M 0s 2024-04-25T03:24:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165550K .......... .......... .......... .......... .......... 98% 24.2M 0s 2024-04-25T03:24:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165600K .......... .......... .......... .......... .......... 98% 24.4M 0s 2024-04-25T03:24:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165650K .......... .......... .......... .......... .......... 98% 35.7M 0s 2024-04-25T03:24:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165700K .......... .......... .......... .......... .......... 98% 22.8M 0s 2024-04-25T03:24:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165750K .......... .......... .......... .......... .......... 98% 25.4M 0s 2024-04-25T03:24:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165800K .......... .......... .......... .......... .......... 98% 44.4M 0s 2024-04-25T03:24:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165850K .......... .......... .......... .......... .......... 98% 23.4M 0s 2024-04-25T03:24:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165900K .......... .......... .......... .......... .......... 98% 47.7M 0s 2024-04-25T03:24:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:165950K .......... .......... .......... .......... .......... 98% 21.8M 0s 2024-04-25T03:24:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166000K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-04-25T03:24:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166050K .......... .......... .......... .......... .......... 98% 53.6M 0s 2024-04-25T03:24:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166100K .......... .......... .......... .......... .......... 98% 22.8M 0s 2024-04-25T03:24:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166150K .......... .......... .......... .......... .......... 98% 9.47M 0s 2024-04-25T03:24:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166200K .......... .......... .......... .......... .......... 98% 108M 0s 2024-04-25T03:24:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166250K .......... .......... .......... .......... .......... 98% 81.9M 0s 2024-04-25T03:24:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166300K .......... .......... .......... .......... .......... 98% 110M 0s 2024-04-25T03:24:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166350K .......... .......... .......... .......... .......... 98% 28.6M 0s 2024-04-25T03:24:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166400K .......... .......... .......... .......... .......... 98% 24.4M 0s 2024-04-25T03:24:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166450K .......... .......... .......... .......... .......... 98% 38.7M 0s 2024-04-25T03:24:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166500K .......... .......... .......... .......... .......... 98% 22.9M 0s 2024-04-25T03:24:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166550K .......... .......... .......... .......... .......... 98% 53.6M 0s 2024-04-25T03:24:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166600K .......... .......... .......... .......... .......... 98% 23.9M 0s 2024-04-25T03:24:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166650K .......... .......... .......... .......... .......... 98% 41.6M 0s 2024-04-25T03:24:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166700K .......... .......... .......... .......... .......... 98% 23.7M 0s 2024-04-25T03:24:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166750K .......... .......... .......... .......... .......... 98% 25.0M 0s 2024-04-25T03:24:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166800K .......... .......... .......... .......... .......... 98% 22.1M 0s 2024-04-25T03:24:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166850K .......... .......... .......... .......... .......... 99% 55.5M 0s 2024-04-25T03:24:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166900K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-04-25T03:24:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:166950K .......... .......... .......... .......... .......... 99% 35.2M 0s 2024-04-25T03:24:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167000K .......... .......... .......... .......... .......... 99% 25.9M 0s 2024-04-25T03:24:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167050K .......... .......... .......... .......... .......... 99% 50.1M 0s 2024-04-25T03:24:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167100K .......... .......... .......... .......... .......... 99% 21.0M 0s 2024-04-25T03:24:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167150K .......... .......... .......... .......... .......... 99% 26.1M 0s 2024-04-25T03:24:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167200K .......... .......... .......... .......... .......... 99% 40.8M 0s 2024-04-25T03:24:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167250K .......... .......... .......... .......... .......... 99% 25.0M 0s 2024-04-25T03:24:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167300K .......... .......... .......... .......... .......... 99% 46.1M 0s 2024-04-25T03:24:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167350K .......... .......... .......... .......... .......... 99% 20.2M 0s 2024-04-25T03:24:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167400K .......... .......... .......... .......... .......... 99% 37.1M 0s 2024-04-25T03:24:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167450K .......... .......... .......... .......... .......... 99% 21.9M 0s 2024-04-25T03:24:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167500K .......... .......... .......... .......... .......... 99% 25.5M 0s 2024-04-25T03:24:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167550K .......... .......... .......... .......... .......... 99% 9.09M 0s 2024-04-25T03:24:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167600K .......... .......... .......... .......... .......... 99% 47.9M 0s 2024-04-25T03:24:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167650K .......... .......... .......... .......... .......... 99% 34.8M 0s 2024-04-25T03:24:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167700K .......... .......... .......... .......... .......... 99% 25.2M 0s 2024-04-25T03:24:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167750K .......... .......... .......... .......... .......... 99% 26.8M 0s 2024-04-25T03:24:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167800K .......... .......... .......... .......... .......... 99% 42.9M 0s 2024-04-25T03:24:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167850K .......... .......... .......... .......... .......... 99% 24.8M 0s 2024-04-25T03:24:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167900K .......... .......... .......... .......... .......... 99% 32.1M 0s 2024-04-25T03:24:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:167950K .......... .......... .......... .......... .......... 99% 27.5M 0s 2024-04-25T03:24:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168000K .......... .......... .......... .......... .......... 99% 23.5M 0s 2024-04-25T03:24:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168050K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-04-25T03:24:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168100K .......... .......... .......... .......... .......... 99% 43.5M 0s 2024-04-25T03:24:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168150K .......... .......... .......... .......... .......... 99% 30.8M 0s 2024-04-25T03:24:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168200K .......... .......... .......... .......... .......... 99% 30.2M 0s 2024-04-25T03:24:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168250K .......... .......... .......... .......... .......... 99% 23.0M 0s 2024-04-25T03:24:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168300K .......... .......... .......... .......... .......... 99% 50.5M 0s 2024-04-25T03:24:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168350K .......... .......... .......... .......... .......... 99% 25.4M 0s 2024-04-25T03:24:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168400K .......... .......... .......... .......... .......... 99% 22.5M 0s 2024-04-25T03:24:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168450K .......... .......... .......... .......... .......... 99% 39.5M 0s 2024-04-25T03:24:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168500K .......... .......... .......... .......... .......... 99% 28.2M 0s 2024-04-25T03:24:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:168550K .......... .......... . 100% 110M=6.0s 2024-04-25T03:24:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:24:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:2024-04-25 03:24:22 (27.3 MB/s) - ‘linux-5.4.tar.gz’ saved [172616875/172616875] 2024-04-25T03:24:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: 2024-04-25T03:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T03:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T03:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T03:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T03:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T03:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T03:24:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T03:24:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T03:24:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T03:24:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T03:24:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T03:24:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 10946a722947 2024-04-25T03:24:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.10946a722947 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:22 2024 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:22 2024 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:22 2024 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 71bc0625-74cf-4a9d-8d49-c65ea91672aa 2024-04-25T03:24:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T03:24:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T03:24:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T03:24:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 714375f0-5687-4663-8a16-5255c33a0cee 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:23.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:23.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T03:24:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T03:24:23.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T03:24:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T03:24:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:24:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:23.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T03:24:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T03:24:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 4 2024-04-25T03:24:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 10a4db33f232 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.10a4db33f232 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:23 2024 2024-04-25T03:24:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:23 2024 2024-04-25T03:24:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:23 2024 2024-04-25T03:24:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 19b060b8-1aa5-4e9c-99e3-6af0d81b59f7 2024-04-25T03:24:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T03:24:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T03:24:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T03:24:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ac4b3a46-ccbe-4aa3-a654-2d3ce1fec293 2024-04-25T03:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:25.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T03:24:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T03:24:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T03:24:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T03:24:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T03:24:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:25.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T03:24:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 6 2024-04-25T03:24:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T03:24:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T03:24:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 10b33716b6a 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.10b33716b6a 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:25 2024 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:25 2024 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:25 2024 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 1b60e7ba-2057-45f0-a4d2-323d67861500 2024-04-25T03:24:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T03:24:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T03:24:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:26.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T03:24:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: aec3d08f-f708-4491-89f4-c090e140731e 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T03:24:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T03:24:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T03:24:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T03:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T03:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T03:24:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 8 2024-04-25T03:24:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T03:24:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T03:24:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 10c2ba37ddbd 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.10c2ba37ddbd 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:26 2024 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:26 2024 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:26 2024 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 11a41978-abcf-4e00-8b6a-d64f8275e921 2024-04-25T03:24:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T03:24:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T03:24:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T03:24:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 4b6a1c3f-005a-454f-98c7-96da0c42acd0 2024-04-25T03:24:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:27.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T03:24:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T03:24:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T03:24:27.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T03:24:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T03:24:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T03:24:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 10 2024-04-25T03:24:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T03:24:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T03:24:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 10d15299c0d3 2024-04-25T03:24:27.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.10d15299c0d3 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:27 2024 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:27 2024 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:27 2024 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: bc92edab-f809-4aea-9d12-81a0e887a025 2024-04-25T03:24:27.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T03:24:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T03:24:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T03:24:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 61fa1f8a-6637-48dd-93e3-47300c1b2efc 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T03:24:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T03:24:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T03:24:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T03:24:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T03:24:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T03:24:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 13 2024-04-25T03:24:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 12 2024-04-25T03:24:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T03:24:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 10e370c76e20 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.10e370c76e20 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:28 2024 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:28 2024 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:28 2024 2024-04-25T03:24:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:24:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T03:24:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T03:24:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T03:24:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 1c446a40-8bb7-434e-90d4-859cff9c46ee 2024-04-25T03:24:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T03:24:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T03:24:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:29.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T03:24:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T03:24:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T03:24:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T03:24:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T03:24:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T03:24:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 16 2024-04-25T03:24:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 15 2024-04-25T03:24:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T03:24:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T03:24:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 10f4c1856eb5 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.10f4c1856eb5 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:29 2024 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:29 2024 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:29 2024 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf 2024-04-25T03:24:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:29.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T03:24:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T03:24:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T03:24:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: d23db3d7-59a0-47c0-b994-97ff0e666eeb 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T03:24:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T03:24:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T03:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T03:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T03:24:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T03:24:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T03:24:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:30.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T03:24:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T03:24:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 19 2024-04-25T03:24:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 18 2024-04-25T03:24:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T03:24:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T03:24:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 11068dc6af4d 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.11068dc6af4d 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:30 2024 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:30 2024 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:30 2024 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: b6294e08-9802-4765-b364-a61b9c32ef66 2024-04-25T03:24:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T03:24:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T03:24:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T03:24:31.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: d8787b63-e8ee-447a-8735-c6f5b736d920 2024-04-25T03:24:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:31.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T03:24:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T03:24:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:31.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T03:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T03:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T03:24:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T03:24:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T03:24:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T03:24:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 22 2024-04-25T03:24:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 21 2024-04-25T03:24:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T03:24:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 11181b00e81a 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.11181b00e81a 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:31 2024 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:31 2024 2024-04-25T03:24:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:31 2024 2024-04-25T03:24:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 115592e3-bacf-4fbc-a20f-0d18299f356e 2024-04-25T03:24:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T03:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T03:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T03:24:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 8badc38d-9830-4263-9b3b-d34c514650d9 2024-04-25T03:24:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:32.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:32.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:32.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T03:24:32.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T03:24:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:24:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:24:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T03:24:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T03:24:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:24:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 26 2024-04-25T03:24:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 25 2024-04-25T03:24:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 24 2024-04-25T03:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T03:24:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 112d974c84fc 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.112d974c84fc 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:24:32 2024 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:24:32 2024 2024-04-25T03:24:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:24:32 2024 2024-04-25T03:24:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:24:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:24:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ab26fb1e-6014-4ff0-91b1-488af1087db4 2024-04-25T03:24:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:24:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:24:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:24:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T03:24:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T03:24:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:24:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T03:24:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 90941299-ba31-4900-b8d7-636b8c1764af 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:24:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:24:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:24:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:24:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:24:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T03:24:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T03:24:33.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:24:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T03:24:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:24:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42334 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T03:24:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T03:24:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:24:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T03:24:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 28 2024-04-25T03:24:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 29 2024-04-25T03:24:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 30 2024-04-25T03:24:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:24:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 31 2024-04-25T03:24:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 32 2024-04-25T03:24:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 33 2024-04-25T03:24:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 34 2024-04-25T03:24:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:24:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:24:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 36 2024-04-25T03:24:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 35 2024-04-25T03:24:36.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 37 2024-04-25T03:24:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:36.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:36.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:24:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:24:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 40 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:37.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 39 2024-04-25T03:24:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:37.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 38 2024-04-25T03:24:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:24:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:24:38.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:38.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 42 2024-04-25T03:24:38.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 41 2024-04-25T03:24:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:24:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:24:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 44 2024-04-25T03:24:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 43 2024-04-25T03:24:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:39.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 45 2024-04-25T03:24:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 47 2024-04-25T03:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:24:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 46 2024-04-25T03:24:41.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:41.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:42.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 49 2024-04-25T03:24:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 48 2024-04-25T03:24:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 50 2024-04-25T03:24:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:24:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:24:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 52 2024-04-25T03:24:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 51 2024-04-25T03:24:43.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 54 2024-04-25T03:24:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:24:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:24:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 53 2024-04-25T03:24:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 56 2024-04-25T03:24:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 57 2024-04-25T03:24:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:24:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:24:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 59 2024-04-25T03:24:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 60 2024-04-25T03:24:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:46.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:47.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 55 2024-04-25T03:24:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:24:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:24:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:47.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 62 2024-04-25T03:24:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 58 2024-04-25T03:24:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 63 2024-04-25T03:24:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:24:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:24:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 61 2024-04-25T03:24:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 65 2024-04-25T03:24:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:24:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:24:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:49.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 64 2024-04-25T03:24:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 67 2024-04-25T03:24:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 66 2024-04-25T03:24:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:24:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:24:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 69 2024-04-25T03:24:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:52.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 68 2024-04-25T03:24:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:24:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:24:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 71 2024-04-25T03:24:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 70 2024-04-25T03:24:52.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 73 2024-04-25T03:24:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:24:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:24:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 72 2024-04-25T03:24:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 74 2024-04-25T03:24:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 75 2024-04-25T03:24:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 76 2024-04-25T03:24:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 77 2024-04-25T03:24:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:24:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:24:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:24:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 78 2024-04-25T03:24:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 79 2024-04-25T03:24:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:24:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:24:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:24:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:24:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:24:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 81 2024-04-25T03:24:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:24:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:24:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:24:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:24:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:24:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:24:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 80 2024-04-25T03:24:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:24:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:24:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:24:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 83 2024-04-25T03:24:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:24:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:24:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:24:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 82 2024-04-25T03:24:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:24:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:24:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:24:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:24:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:24:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:24:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 84 2024-04-25T03:24:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 85 2024-04-25T03:24:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:24:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:24:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:24:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:24:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:24:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:24:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:24:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 86 2024-04-25T03:25:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:00.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 87 2024-04-25T03:25:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:00.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 89 2024-04-25T03:25:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 88 2024-04-25T03:25:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 91 2024-04-25T03:25:01.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:02.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 90 2024-04-25T03:25:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 93 2024-04-25T03:25:02.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:02.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 92 2024-04-25T03:25:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 94 2024-04-25T03:25:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 95 2024-04-25T03:25:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 96 2024-04-25T03:25:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 97 2024-04-25T03:25:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:04.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 99 2024-04-25T03:25:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:05.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 98 2024-04-25T03:25:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:06.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 101 2024-04-25T03:25:06.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 100 2024-04-25T03:25:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 102 2024-04-25T03:25:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:07.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 103 2024-04-25T03:25:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 105 2024-04-25T03:25:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 104 2024-04-25T03:25:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:09.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:09.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 106 2024-04-25T03:25:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 107 2024-04-25T03:25:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 108 2024-04-25T03:25:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 109 2024-04-25T03:25:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:11.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 111 2024-04-25T03:25:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 110 2024-04-25T03:25:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:13.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 112 2024-04-25T03:25:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 113 2024-04-25T03:25:13.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 115 2024-04-25T03:25:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 114 2024-04-25T03:25:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 116 2024-04-25T03:25:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 117 2024-04-25T03:25:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:14.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 119 2024-04-25T03:25:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 118 2024-04-25T03:25:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:16.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 121 2024-04-25T03:25:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 120 2024-04-25T03:25:17.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 123 2024-04-25T03:25:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 122 2024-04-25T03:25:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 125 2024-04-25T03:25:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:19.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 124 2024-04-25T03:25:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:20.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:20.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 127 2024-04-25T03:25:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 126 2024-04-25T03:25:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 128 2024-04-25T03:25:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 129 2024-04-25T03:25:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 131 2024-04-25T03:25:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 130 2024-04-25T03:25:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 133 2024-04-25T03:25:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 132 2024-04-25T03:25:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:23.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:25:23.914+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665681400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:25:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 135 2024-04-25T03:25:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 134 2024-04-25T03:25:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 137 2024-04-25T03:25:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 136 2024-04-25T03:25:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 138 2024-04-25T03:25:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 139 2024-04-25T03:25:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 141 2024-04-25T03:25:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:27.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 140 2024-04-25T03:25:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 142 2024-04-25T03:25:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 143 2024-04-25T03:25:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 144 2024-04-25T03:25:28.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 145 2024-04-25T03:25:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 146 2024-04-25T03:25:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 147 2024-04-25T03:25:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:30.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 148 2024-04-25T03:25:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 149 2024-04-25T03:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:30.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 151 2024-04-25T03:25:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 150 2024-04-25T03:25:32.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 155 2024-04-25T03:25:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 152 2024-04-25T03:25:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 153 2024-04-25T03:25:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:33.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 154 2024-04-25T03:25:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 156 2024-04-25T03:25:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 157 2024-04-25T03:25:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:35.836 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:25:35.834+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664daa000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:25:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 158 2024-04-25T03:25:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 159 2024-04-25T03:25:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 162 2024-04-25T03:25:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 161 2024-04-25T03:25:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 160 2024-04-25T03:25:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 164 2024-04-25T03:25:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 163 2024-04-25T03:25:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:37.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 165 2024-04-25T03:25:38.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 166 2024-04-25T03:25:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:39.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 167 2024-04-25T03:25:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:40.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 168 2024-04-25T03:25:40.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 170 2024-04-25T03:25:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 169 2024-04-25T03:25:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:40.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:41.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 172 2024-04-25T03:25:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:42.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 171 2024-04-25T03:25:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 173 2024-04-25T03:25:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 174 2024-04-25T03:25:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 175 2024-04-25T03:25:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 179 2024-04-25T03:25:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 177 2024-04-25T03:25:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 178 2024-04-25T03:25:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 176 2024-04-25T03:25:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:45.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 180 2024-04-25T03:25:46.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:25:46.872+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664cc2800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:25:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 182 2024-04-25T03:25:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 181 2024-04-25T03:25:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 183 2024-04-25T03:25:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 184 2024-04-25T03:25:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 185 2024-04-25T03:25:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:49.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:25:49.156+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665735800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:25:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 187 2024-04-25T03:25:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 188 2024-04-25T03:25:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 186 2024-04-25T03:25:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 189 2024-04-25T03:25:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 190 2024-04-25T03:25:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 192 2024-04-25T03:25:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 191 2024-04-25T03:25:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 193 2024-04-25T03:25:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 194 2024-04-25T03:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:25:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:25:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:25:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 195 2024-04-25T03:25:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 196 2024-04-25T03:25:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 197 2024-04-25T03:25:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 198 2024-04-25T03:25:54.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:25:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:25:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:25:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:25:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 201 2024-04-25T03:25:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 200 2024-04-25T03:25:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:25:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:25:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:25:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:25:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:25:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:25:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:25:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:25:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 202 2024-04-25T03:25:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:57.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 204 2024-04-25T03:25:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 206 2024-04-25T03:25:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:25:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:25:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:25:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:25:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:25:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:25:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 203 2024-04-25T03:25:58.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:58.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:25:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:25:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:25:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 199 2024-04-25T03:25:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 205 2024-04-25T03:25:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 209 2024-04-25T03:25:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:25:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:25:59.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:25:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:25:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:25:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:25:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:25:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:25:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:25:59.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 207 2024-04-25T03:26:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 208 2024-04-25T03:26:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 210 2024-04-25T03:26:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 211 2024-04-25T03:26:02.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 213 2024-04-25T03:26:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 214 2024-04-25T03:26:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 212 2024-04-25T03:26:03.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:03.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 217 2024-04-25T03:26:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 215 2024-04-25T03:26:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 216 2024-04-25T03:26:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 219 2024-04-25T03:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 218 2024-04-25T03:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 220 2024-04-25T03:26:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:06.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 221 2024-04-25T03:26:06.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 222 2024-04-25T03:26:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 224 2024-04-25T03:26:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 225 2024-04-25T03:26:08.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 223 2024-04-25T03:26:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:08.911 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:08.909+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596659a8800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 227 2024-04-25T03:26:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 226 2024-04-25T03:26:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 228 2024-04-25T03:26:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 229 2024-04-25T03:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:10.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 230 2024-04-25T03:26:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 231 2024-04-25T03:26:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 232 2024-04-25T03:26:12.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:12.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 235 2024-04-25T03:26:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 234 2024-04-25T03:26:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 233 2024-04-25T03:26:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 236 2024-04-25T03:26:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 237 2024-04-25T03:26:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:14.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:14.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 239 2024-04-25T03:26:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 238 2024-04-25T03:26:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 240 2024-04-25T03:26:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:16.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:16.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 241 2024-04-25T03:26:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 242 2024-04-25T03:26:17.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 243 2024-04-25T03:26:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 244 2024-04-25T03:26:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 246 2024-04-25T03:26:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 245 2024-04-25T03:26:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:18.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 247 2024-04-25T03:26:18.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 248 2024-04-25T03:26:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 249 2024-04-25T03:26:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 250 2024-04-25T03:26:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 251 2024-04-25T03:26:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 252 2024-04-25T03:26:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 254 2024-04-25T03:26:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:22.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 257 2024-04-25T03:26:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 255 2024-04-25T03:26:24.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 253 2024-04-25T03:26:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 258 2024-04-25T03:26:25.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 259 2024-04-25T03:26:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:25.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 260 2024-04-25T03:26:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 256 2024-04-25T03:26:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 261 2024-04-25T03:26:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 262 2024-04-25T03:26:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 263 2024-04-25T03:26:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 265 2024-04-25T03:26:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 264 2024-04-25T03:26:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 267 2024-04-25T03:26:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 268 2024-04-25T03:26:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 269 2024-04-25T03:26:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 266 2024-04-25T03:26:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 271 2024-04-25T03:26:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 270 2024-04-25T03:26:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:32.848 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:32.846+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664daa400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:32.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:32.866+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596653ef400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 272 2024-04-25T03:26:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 273 2024-04-25T03:26:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 274 2024-04-25T03:26:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 275 2024-04-25T03:26:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 276 2024-04-25T03:26:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 277 2024-04-25T03:26:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 278 2024-04-25T03:26:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 279 2024-04-25T03:26:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:36.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 280 2024-04-25T03:26:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 281 2024-04-25T03:26:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:36.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 283 2024-04-25T03:26:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 282 2024-04-25T03:26:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:38.835 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:38.831+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665599000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 285 2024-04-25T03:26:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 284 2024-04-25T03:26:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:39.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:39.873+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596655a8000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 286 2024-04-25T03:26:40.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 287 2024-04-25T03:26:40.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:40.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 288 2024-04-25T03:26:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 289 2024-04-25T03:26:41.876 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:41.872+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664ec7000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:41.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:41.890+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966573cc00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 293 2024-04-25T03:26:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 292 2024-04-25T03:26:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 290 2024-04-25T03:26:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 291 2024-04-25T03:26:42.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:42.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 295 2024-04-25T03:26:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 294 2024-04-25T03:26:43.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:26:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:26:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:26:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:26:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:26:45.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 297 2024-04-25T03:26:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 296 2024-04-25T03:26:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:26:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:26:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:45.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 298 2024-04-25T03:26:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 301 2024-04-25T03:26:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 299 2024-04-25T03:26:45.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 300 2024-04-25T03:26:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:26:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:26:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 303 2024-04-25T03:26:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 302 2024-04-25T03:26:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 305 2024-04-25T03:26:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 304 2024-04-25T03:26:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:26:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:26:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:26:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:26:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 307 2024-04-25T03:26:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42476 2024-04-25T03:26:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 306 2024-04-25T03:26:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:26:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:26:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:26:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42610 2024-04-25T03:26:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42744 2024-04-25T03:26:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:26:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:26:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:26:50.897 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:26:50.894+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966524d400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:26:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42898 2024-04-25T03:26:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 308 2024-04-25T03:26:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 309 2024-04-25T03:26:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:26:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:26:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:26:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:26:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:26:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:26:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43055 2024-04-25T03:26:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 310 2024-04-25T03:26:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 311 2024-04-25T03:26:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43210 2024-04-25T03:26:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 312 2024-04-25T03:26:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:26:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43365 2024-04-25T03:26:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43540 2024-04-25T03:26:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:26:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43716 2024-04-25T03:26:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:26:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:26:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42335 2024-04-25T03:29:24.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42477 2024-04-25T03:29:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42611 2024-04-25T03:29:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42745 2024-04-25T03:29:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 42899 2024-04-25T03:29:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43056 2024-04-25T03:29:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43211 2024-04-25T03:29:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43366 2024-04-25T03:29:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43541 2024-04-25T03:29:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:29:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 43717 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T03:29:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T03:29:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T03:29:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T03:29:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T03:29:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T03:29:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T03:29:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T03:29:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:29:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49716 2024-04-25T03:29:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T03:29:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T03:29:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T03:29:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T03:29:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:29:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T03:29:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:29:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:29:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:29:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:29:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:29:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:29:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:29:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:29:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:29:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:29:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:29:34.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=105208ef2b48ad101fa9b3b216ca6d62 2024-04-25T03:30:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T03:30:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=62018df7a4d0b6cba42f0f40ff15710f 2024-04-25T03:30:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T03:30:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=289679dcabd0e3d5f77f7969fb4bfaa1 2024-04-25T03:30:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T03:30:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T03:30:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T03:30:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T03:30:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T03:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T03:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T03:30:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=161d714fd732997925b94b880e6c11ba 2024-04-25T03:30:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T03:30:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T03:30:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T03:30:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=eb80f479fac94606c4ca5c999fbe1169 2024-04-25T03:30:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T03:30:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=b61c2aad2e1b05dd0618990b39f6143c 2024-04-25T03:30:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T03:30:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=8ed8a8877a255bef51db87ac197bffd5 2024-04-25T03:30:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T03:30:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T03:30:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T03:30:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T03:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T03:30:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T03:30:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T03:30:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:31.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T03:30:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=863392895af172a76f8a519216ddc898 2024-04-25T03:30:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T03:30:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=962f0b65715f0e6ae0479ea2b43be613 2024-04-25T03:30:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T03:30:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T03:30:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T03:30:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T03:30:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T03:30:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=177f12f218a67d7fae5d8be43fd6ef4a 2024-04-25T03:30:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T03:30:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T03:30:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:31.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T03:30:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:30:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:30:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T03:30:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:32.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T03:30:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:30:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:30:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:32.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:32.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T03:30:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:30:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:30:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:30:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:30:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:30:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:30:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:33.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:30:33.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:30:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:30:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:30:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 11a41978-abcf-4e00-8b6a-d64f8275e921 2024-04-25T03:30:33.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:33.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:30:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab26fb1e-6014-4ff0-91b1-488af1087db4 2024-04-25T03:30:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 115592e3-bacf-4fbc-a20f-0d18299f356e 2024-04-25T03:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:30:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 19b060b8-1aa5-4e9c-99e3-6af0d81b59f7 2024-04-25T03:30:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:30:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf 2024-04-25T03:30:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:30:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:30:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:30:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:30:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b6294e08-9802-4765-b364-a61b9c32ef66 2024-04-25T03:30:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:30:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 71bc0625-74cf-4a9d-8d49-c65ea91672aa 2024-04-25T03:30:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:30:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bc92edab-f809-4aea-9d12-81a0e887a025 2024-04-25T03:30:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:30:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:30:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:30:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:30:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:35.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:30:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:30:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:30:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 11a41978-abcf-4e00-8b6a-d64f8275e921 2024-04-25T03:30:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:35.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:35.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:30:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:30:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab26fb1e-6014-4ff0-91b1-488af1087db4 2024-04-25T03:30:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:30:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:30:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:30:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 115592e3-bacf-4fbc-a20f-0d18299f356e 2024-04-25T03:30:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:30:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 19b060b8-1aa5-4e9c-99e3-6af0d81b59f7 2024-04-25T03:30:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:30:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf 2024-04-25T03:30:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:35.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:30:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1b60e7ba-2057-45f0-a4d2-323d67861500 2024-04-25T03:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:30:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:30:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:30:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:30:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:36.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:30:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:30:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b6294e08-9802-4765-b364-a61b9c32ef66 2024-04-25T03:30:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:30:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bc92edab-f809-4aea-9d12-81a0e887a025 2024-04-25T03:30:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 71bc0625-74cf-4a9d-8d49-c65ea91672aa 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:30:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:30:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1b60e7ba-2057-45f0-a4d2-323d67861500 2024-04-25T03:30:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:30:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:30:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:30:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:30:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:30:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:30:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:30:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:30:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:30:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 11a41978-abcf-4e00-8b6a-d64f8275e921 2024-04-25T03:30:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:30:39.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:30:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:30:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:30:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:30:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab26fb1e-6014-4ff0-91b1-488af1087db4 2024-04-25T03:30:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:39.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:30:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:30:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 115592e3-bacf-4fbc-a20f-0d18299f356e 2024-04-25T03:30:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 19b060b8-1aa5-4e9c-99e3-6af0d81b59f7 2024-04-25T03:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:30:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf 2024-04-25T03:30:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:30:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:30:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:30:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:30:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:30:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b6294e08-9802-4765-b364-a61b9c32ef66 2024-04-25T03:30:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:40.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:30:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bc92edab-f809-4aea-9d12-81a0e887a025 2024-04-25T03:30:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 71bc0625-74cf-4a9d-8d49-c65ea91672aa 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:40.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:30:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:30:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:30:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1b60e7ba-2057-45f0-a4d2-323d67861500 2024-04-25T03:30:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:30:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:30:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:30:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:30:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:30:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:16' 2024-04-25T03:30:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:46.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:30:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:30:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:30:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:47.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:30:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:30:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 11a41978-abcf-4e00-8b6a-d64f8275e921 2024-04-25T03:30:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:30:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:30:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:30:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab26fb1e-6014-4ff0-91b1-488af1087db4 2024-04-25T03:30:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:30:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:30:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:47.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:30:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 115592e3-bacf-4fbc-a20f-0d18299f356e 2024-04-25T03:30:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:47.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:47.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:30:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 19b060b8-1aa5-4e9c-99e3-6af0d81b59f7 2024-04-25T03:30:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:47.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:30:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf 2024-04-25T03:30:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:47.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:30:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:30:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:30:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:30:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:30:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:30:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:30:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:30:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T03:30:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:30:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:30:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:30:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:30:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:30:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b6294e08-9802-4765-b364-a61b9c32ef66 2024-04-25T03:30:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 71bc0625-74cf-4a9d-8d49-c65ea91672aa 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bc92edab-f809-4aea-9d12-81a0e887a025 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:48.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:30:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:30:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:30:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:30:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:30:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T03:30:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T03:30:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T03:30:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T03:30:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:30:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab26fb1e-6014-4ff0-91b1-488af1087db4 2024-04-25T03:30:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:30:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 11a41978-abcf-4e00-8b6a-d64f8275e921 2024-04-25T03:30:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 115592e3-bacf-4fbc-a20f-0d18299f356e 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 19b060b8-1aa5-4e9c-99e3-6af0d81b59f7 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:30:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf 2024-04-25T03:30:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:30:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:30:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:30:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:30:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T03:30:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:30:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T03:30:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:30:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:30:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T03:30:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:30:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T03:30:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:30:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:30:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T03:30:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:30:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T03:30:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T03:30:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:30:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:30:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T03:30:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T03:30:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T03:30:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:30:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b6294e08-9802-4765-b364-a61b9c32ef66 2024-04-25T03:30:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:30:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 71bc0625-74cf-4a9d-8d49-c65ea91672aa 2024-04-25T03:30:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:30:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bc92edab-f809-4aea-9d12-81a0e887a025 2024-04-25T03:30:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:30:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:30:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T03:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T03:30:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:30:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:30:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:30:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:30:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T03:30:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:30:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T03:30:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:49.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T03:30:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T03:30:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T03:30:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T03:30:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T03:30:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T03:30:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T03:30:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:30:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T03:30:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:30:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T03:30:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T03:30:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:30:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:30:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T03:30:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:30:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:30:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:30:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:30:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1b60e7ba-2057-45f0-a4d2-323d67861500 2024-04-25T03:30:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:30:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:30:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T03:30:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T03:30:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T03:30:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T03:30:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T03:30:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:30:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:30:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T03:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:30:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T03:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T03:30:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:30:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1b60e7ba-2057-45f0-a4d2-323d67861500 2024-04-25T03:30:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:30:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:30:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T03:30:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T03:30:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T03:30:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:30:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:30:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:30:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:30:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:30:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:30:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:30:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:30:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:30:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T03:30:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:30:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:30:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T03:30:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:30:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:30:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:30:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:30:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:30:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:30:46' 2024-04-25T03:30:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:30:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:30:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:30:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:30:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:56.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:30:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:30:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:30:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:30:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:30:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 03:30:46' 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:30:46' 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:30:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:30:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:30:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:30:46' 2024-04-25T03:30:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:30:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:30:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:30:46' 2024-04-25T03:30:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:30:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:30:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:30:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:30:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:30:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T03:30:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:30:57.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:30:57.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:30:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T03:30:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T03:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:30:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:31:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=62018df7a4d0b6cba42f0f40ff15710f 2024-04-25T03:31:44.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T03:31:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 62018df7a4d0b6cba42f0f40ff15710f != \6\2\0\1\8\d\f\7\a\4\d\0\b\6\c\b\a\4\2\f\0\f\4\0\f\f\1\5\7\1\0\f ]] 2024-04-25T03:31:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=863392895af172a76f8a519216ddc898 2024-04-25T03:31:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T03:31:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 863392895af172a76f8a519216ddc898 != \8\6\3\3\9\2\8\9\5\a\f\1\7\2\a\7\6\f\8\a\5\1\9\2\1\6\d\d\c\8\9\8 ]] 2024-04-25T03:31:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=962f0b65715f0e6ae0479ea2b43be613 2024-04-25T03:31:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T03:31:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 962f0b65715f0e6ae0479ea2b43be613 != \9\6\2\f\0\b\6\5\7\1\5\f\0\e\6\a\e\0\4\7\9\e\a\2\b\4\3\b\e\6\1\3 ]] 2024-04-25T03:31:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=8ed8a8877a255bef51db87ac197bffd5 2024-04-25T03:31:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T03:31:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=105208ef2b48ad101fa9b3b216ca6d62 2024-04-25T03:31:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T03:31:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=b61c2aad2e1b05dd0618990b39f6143c 2024-04-25T03:31:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T03:31:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=eb80f479fac94606c4ca5c999fbe1169 2024-04-25T03:31:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T03:31:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ b61c2aad2e1b05dd0618990b39f6143c != \b\6\1\c\2\a\a\d\2\e\1\b\0\5\d\d\0\6\1\8\9\9\0\b\3\9\f\6\1\4\3\c ]] 2024-04-25T03:31:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 105208ef2b48ad101fa9b3b216ca6d62 != \1\0\5\2\0\8\e\f\2\b\4\8\a\d\1\0\1\f\a\9\b\3\b\2\1\6\c\a\6\d\6\2 ]] 2024-04-25T03:31:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 8ed8a8877a255bef51db87ac197bffd5 != \8\e\d\8\a\8\8\7\7\a\2\5\5\b\e\f\5\1\d\b\8\7\a\c\1\9\7\b\f\f\d\5 ]] 2024-04-25T03:31:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49717 2024-04-25T03:31:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49719 2024-04-25T03:31:47.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ eb80f479fac94606c4ca5c999fbe1169 != \e\b\8\0\f\4\7\9\f\a\c\9\4\6\0\6\c\4\c\a\5\c\9\9\9\f\b\e\1\1\6\9 ]] 2024-04-25T03:31:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=289679dcabd0e3d5f77f7969fb4bfaa1 2024-04-25T03:31:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T03:31:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 289679dcabd0e3d5f77f7969fb4bfaa1 != \2\8\9\6\7\9\d\c\a\b\d\0\e\3\d\5\f\7\7\f\7\9\6\9\f\b\4\b\f\a\a\1 ]] 2024-04-25T03:31:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=177f12f218a67d7fae5d8be43fd6ef4a 2024-04-25T03:31:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T03:31:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 177f12f218a67d7fae5d8be43fd6ef4a != \1\7\7\f\1\2\f\2\1\8\a\6\7\d\7\f\a\e\5\d\8\b\e\4\3\f\d\6\e\f\4\a ]] 2024-04-25T03:31:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49721 2024-04-25T03:31:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49723 2024-04-25T03:31:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49725 2024-04-25T03:31:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=161d714fd732997925b94b880e6c11ba 2024-04-25T03:31:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T03:31:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 161d714fd732997925b94b880e6c11ba != \1\6\1\d\7\1\4\f\d\7\3\2\9\9\7\9\2\5\b\9\4\b\8\8\0\e\6\c\1\1\b\a ]] 2024-04-25T03:31:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49727 2024-04-25T03:31:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49730 2024-04-25T03:31:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49731 2024-04-25T03:31:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 49733 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:31:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:31:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:31:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:31:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:31:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T03:31:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:31:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:31:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:31:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:31:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:31:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 71bc0625-74cf-4a9d-8d49-c65ea91672aa 2024-04-25T03:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:31:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3603660.8,"bytes_per_snapshot":72073216.0,"last_snapshot_bytes":72073216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"} 2024-04-25T03:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:31:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:31:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:31:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:31:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3603660.8,"bytes_per_snapshot":72073216.0,"last_snapshot_bytes":72073216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"} 2024-04-25T03:31:52.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:31:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:31:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:31:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:31:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:31:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:31:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:31:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:31:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:31:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:31:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:31:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary171bc0625-74cf-4a9d-8d49-c65ea91672aaup+replayingreplaying, {"bytes_per_second":3603660.8,"bytes_per_snapshot":72073216.0,"last_snapshot_bytes":72073216,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+replayingreplaying, {"bytes_per_second":3272908.8,"bytes_per_snapshot":65458176.0,"last_snapshot_bytes":65458176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+replayingreplaying, {"bytes_per_second":3266969.6,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015852,"remote_snapshot_timestamp":1714015852,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+replayingreplaying, {"bytes_per_second":3641548.8,"bytes_per_snapshot":72830976.0,"last_snapshot_bytes":72830976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+replayingreplaying, {"bytes_per_second":3617587.2,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+replayingreplaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+replayingreplaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:31:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:31:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:31:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:31:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:31:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:31:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:31:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:31:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:31:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T03:31:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:31:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:31:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T03:31:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T03:31:52.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:31:52.977+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c63040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:31:52.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:31:52.977+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/71bc0625-74cf-4a9d-8d49-c65ea91672aa] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:31:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:31:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T03:31:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:31:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:31:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:31:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T03:31:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T03:31:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:31:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:31:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T03:31:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:31:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:31:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:31:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T03:31:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:31:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:31:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:31:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:31:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:31:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:31:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:31:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 19b060b8-1aa5-4e9c-99e3-6af0d81b59f7 2024-04-25T03:31:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:31:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3272908.8,"bytes_per_snapshot":65458176.0,"last_snapshot_bytes":65458176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:31:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:31:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:31:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:31:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:31:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:31:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3272908.8,"bytes_per_snapshot":65458176.0,"last_snapshot_bytes":65458176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:31:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:31:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:31:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:31:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:31:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:31:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:31:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:31:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:31:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:31:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:31:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:31:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:31:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:31:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:31:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:31:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary219b060b8-1aa5-4e9c-99e3-6af0d81b59f7up+replayingreplaying, {"bytes_per_second":3272908.8,"bytes_per_snapshot":65458176.0,"last_snapshot_bytes":65458176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+replayingreplaying, {"bytes_per_second":3266969.6,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015852,"remote_snapshot_timestamp":1714015852,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+replayingreplaying, {"bytes_per_second":3641548.8,"bytes_per_snapshot":72830976.0,"last_snapshot_bytes":72830976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+replayingreplaying, {"bytes_per_second":3617587.2,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+replayingreplaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+replayingreplaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:31:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:31:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:31:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:31:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:31:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:31:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:31:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:31:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:31:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T03:31:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:31:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:31:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T03:31:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T03:31:56.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:31:56.539+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664d101a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:31:56.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:31:56.539+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/19b060b8-1aa5-4e9c-99e3-6af0d81b59f7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:31:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:31:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T03:31:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:31:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:31:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:31:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T03:31:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:31:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:31:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T03:31:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:31:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:31:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:31:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:31:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:31:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:31:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:31:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1b60e7ba-2057-45f0-a4d2-323d67861500 2024-04-25T03:31:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:31:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3266969.6,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015852,"remote_snapshot_timestamp":1714015852,"replay_state":"idle"} 2024-04-25T03:31:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:31:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:31:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:31:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:31:59.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:31:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3266969.6,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015852,"remote_snapshot_timestamp":1714015852,"replay_state":"idle"} 2024-04-25T03:31:59.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:31:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:31:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:31:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:31:59.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:31:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:31:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:31:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:31:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:31:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary31b60e7ba-2057-45f0-a4d2-323d67861500up+replayingreplaying, {"bytes_per_second":3266969.6,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015852,"remote_snapshot_timestamp":1714015852,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+replayingreplaying, {"bytes_per_second":3641548.8,"bytes_per_snapshot":72830976.0,"last_snapshot_bytes":72830976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+replayingreplaying, {"bytes_per_second":3617587.2,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+replayingreplaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+replayingreplaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:31:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:31:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:31:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:31:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:31:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:31:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:31:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:31:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:31:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:31:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T03:31:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:31:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:31:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:31:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T03:31:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T03:31:59.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:31:59.530+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664d10820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:31:59.533 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:31:59.530+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdaa00 [3/1b60e7ba-2057-45f0-a4d2-323d67861500] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T03:32:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T03:32:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:01.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T03:32:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T03:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:32:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:32:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 11a41978-abcf-4e00-8b6a-d64f8275e921 2024-04-25T03:32:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:32:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3641548.8,"bytes_per_snapshot":72830976.0,"last_snapshot_bytes":72830976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:32:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:32:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:32:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:32:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:32:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3641548.8,"bytes_per_snapshot":72830976.0,"last_snapshot_bytes":72830976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:32:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:02.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:32:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:32:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:32:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:32:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:32:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:32:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:32:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:32:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary411a41978-abcf-4e00-8b6a-d64f8275e921up+replayingreplaying, {"bytes_per_second":3641548.8,"bytes_per_snapshot":72830976.0,"last_snapshot_bytes":72830976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+replayingreplaying, {"bytes_per_second":3617587.2,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+replayingreplaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+replayingreplaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:32:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:32:02.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:32:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:32:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:32:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:32:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T03:32:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:32:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T03:32:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T03:32:02.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:02.556+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664d10000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:32:02.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:02.556+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/11a41978-abcf-4e00-8b6a-d64f8275e921] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T03:32:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:04.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:04.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T03:32:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T03:32:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:32:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T03:32:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:32:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:32:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bc92edab-f809-4aea-9d12-81a0e887a025 2024-04-25T03:32:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:32:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3617587.2,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"} 2024-04-25T03:32:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:32:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:32:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:32:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:32:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:32:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3617587.2,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"} 2024-04-25T03:32:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:32:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:32:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:32:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:32:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:32:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:32:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:32:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:32:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary5bc92edab-f809-4aea-9d12-81a0e887a025up+replayingreplaying, {"bytes_per_second":3617587.2,"bytes_per_snapshot":72351744.0,"last_snapshot_bytes":72351744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+replayingreplaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+replayingreplaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:32:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:32:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:32:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:32:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:32:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:05.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:32:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T03:32:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:32:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T03:32:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T03:32:05.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:05.559+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664486000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:32:05.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:05.560+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/bc92edab-f809-4aea-9d12-81a0e887a025] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T03:32:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T03:32:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T03:32:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:32:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T03:32:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:32:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T03:32:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:32:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:32:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54494967-9688-4acd-bd42-3111df2fb71a 2024-04-25T03:32:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:32:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"} 2024-04-25T03:32:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:32:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:32:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:32:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:32:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:32:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"} 2024-04-25T03:32:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:32:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:32:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:32:08.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:32:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:32:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:32:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:32:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary654494967-9688-4acd-bd42-3111df2fb71aup+replayingreplaying, {"bytes_per_second":3685785.6,"bytes_per_snapshot":73715712.0,"last_snapshot_bytes":73715712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015857,"remote_snapshot_timestamp":1714015857,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+replayingreplaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:32:08.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:32:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:32:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:32:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:32:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:32:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T03:32:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:32:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T03:32:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T03:32:08.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:08.464+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664d10d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:32:08.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:08.464+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/54494967-9688-4acd-bd42-3111df2fb71a] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T03:32:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:10.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T03:32:10.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T03:32:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:32:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T03:32:11.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:32:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:32:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf 2024-04-25T03:32:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:32:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:32:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:32:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:32:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:32:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:32:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:32:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:32:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:11.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:32:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:32:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:32:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:32:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:32:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary7c7fd85c9-c0d4-465a-86ce-4e18cfba1eafup+replayingreplaying, {"bytes_per_second":3258368.0,"bytes_per_snapshot":65167360.0,"last_snapshot_bytes":65167360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:32:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:32:11.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:32:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:32:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:32:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:32:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T03:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T03:32:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T03:32:11.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:11.556+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c62680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:32:11.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:11.556+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/c7fd85c9-c0d4-465a-86ce-4e18cfba1eaf] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T03:32:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T03:32:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:13.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:32:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:32:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b6294e08-9802-4765-b364-a61b9c32ef66 2024-04-25T03:32:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:32:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"} 2024-04-25T03:32:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:32:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:31:46' 2024-04-25T03:32:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:32:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:32:14.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:32:14.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"} 2024-04-25T03:32:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:32:14.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:32:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:32:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:32:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:32:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:32:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 03:31:46' 2024-04-25T03:32:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:31:46' 2024-04-25T03:32:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:32:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":3686195.2,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary8b6294e08-9802-4765-b364-a61b9c32ef66up+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015850,"remote_snapshot_timestamp":1714015850,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":3687014.4,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:31:46' 2024-04-25T03:32:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:32:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:32:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:31:46' 2024-04-25T03:32:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:32:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:32:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:32:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T03:32:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:32:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T03:32:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T03:32:14.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:14.395+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664480ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:32:14.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:14.395+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdac80 [3/b6294e08-9802-4765-b364-a61b9c32ef66] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T03:32:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T03:32:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T03:32:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 115592e3-bacf-4fbc-a20f-0d18299f356e 2024-04-25T03:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:32:16' 2024-04-25T03:32:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:32:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:32:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:32:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:32:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:32:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:32:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:32:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:32:17.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:32:16' 2024-04-25T03:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 03:32:16' 2024-04-25T03:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:32:16' 2024-04-25T03:32:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:32:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:32:16image-primary9115592e3-bacf-4fbc-a20f-0d18299f356eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":73740288.0,"last_snapshot_bytes":73740288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:32:16' 2024-04-25T03:32:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:32:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:32:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:32:16' 2024-04-25T03:32:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:32:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:32:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:32:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T03:32:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:32:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T03:32:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T03:32:17.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:17.392+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596644804e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:32:17.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:17.392+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb680 [3/115592e3-bacf-4fbc-a20f-0d18299f356e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T03:32:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T03:32:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:19.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T03:32:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:32:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T03:32:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:32:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:32:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:32:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab26fb1e-6014-4ff0-91b1-488af1087db4 2024-04-25T03:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:32:16' 2024-04-25T03:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:32:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:32:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:32:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"} 2024-04-25T03:32:20.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:32:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:32:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:32:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:32:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:32:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:32:16' 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 03:32:16' 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:32:16' 2024-04-25T03:32:20.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:32:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:32:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:32:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10ab26fb1e-6014-4ff0-91b1-488af1087db4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":73723904.0,"last_snapshot_bytes":73723904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714015849,"remote_snapshot_timestamp":1714015849,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:32:16' 2024-04-25T03:32:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:32:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:32:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:32:16' 2024-04-25T03:32:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:32:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:32:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:32:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:32:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T03:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T03:32:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T03:32:20.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:20.396+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c62000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:32:20.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:20.396+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/ab26fb1e-6014-4ff0-91b1-488af1087db4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:32:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:32:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T03:32:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:32:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:32:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T03:32:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T03:32:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T03:32:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T03:32:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T03:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T03:32:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T03:32:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T03:32:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 158b99e366f6 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.158b99e366f6 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:22 2024 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:22 2024 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:22 2024 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: e3bb2dd3-8ca0-4a47-8072-65e4a79f4950 2024-04-25T03:32:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T03:32:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T03:32:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T03:32:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 43f9a7b3-a68a-40b9-8158-6e2eb6e83eb4 2024-04-25T03:32:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T03:32:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T03:32:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T03:32:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T03:32:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T03:32:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T03:32:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T03:32:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 334 2024-04-25T03:32:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T03:32:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 159b7f6e9ee7 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.159b7f6e9ee7 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:23 2024 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:23 2024 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:23 2024 2024-04-25T03:32:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4 2024-04-25T03:32:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:23.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T03:32:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T03:32:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T03:32:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:24.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 46bf7f36-43c9-41be-8bcb-109dd58d1c86 2024-04-25T03:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T03:32:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T03:32:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T03:32:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T03:32:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T03:32:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T03:32:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T03:32:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T03:32:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T03:32:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T03:32:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 336 2024-04-25T03:32:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T03:32:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T03:32:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 15aa78ca363f 2024-04-25T03:32:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.15aa78ca363f 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:24 2024 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:24 2024 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:24 2024 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:32:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T03:32:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T03:32:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T03:32:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: c8688fd5-ce46-4f2e-9649-bc0c954e86e2 2024-04-25T03:32:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T03:32:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T03:32:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:25.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T03:32:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T03:32:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T03:32:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 338 2024-04-25T03:32:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T03:32:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 15b8d537ea65 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.15b8d537ea65 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:25 2024 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:25 2024 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:25 2024 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: eb5ea9d4-e7be-4bfc-8787-d1c9443393ff 2024-04-25T03:32:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T03:32:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T03:32:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T03:32:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:26.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 6942595b-7c35-4a68-b5ee-8f6fa5f02a61 2024-04-25T03:32:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:26.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T03:32:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T03:32:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T03:32:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T03:32:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T03:32:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T03:32:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T03:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T03:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T03:32:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T03:32:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 340 2024-04-25T03:32:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T03:32:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T03:32:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 15c73e25c7dd 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.15c73e25c7dd 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:26 2024 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:26 2024 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:26 2024 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:32:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T03:32:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T03:32:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T03:32:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: cf3662ac-afd8-4a3c-ba98-8b6b80e28a45 2024-04-25T03:32:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T03:32:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T03:32:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T03:32:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T03:32:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:27.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T03:32:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T03:32:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T03:32:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T03:32:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 343 2024-04-25T03:32:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 342 2024-04-25T03:32:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T03:32:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 15d9fe55de41 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.15d9fe55de41 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:27 2024 2024-04-25T03:32:27.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:27 2024 2024-04-25T03:32:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:27 2024 2024-04-25T03:32:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:32:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T03:32:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T03:32:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T03:32:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: d6f17ffc-9cac-4f7a-8eff-a0e0a6162791 2024-04-25T03:32:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T03:32:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T03:32:28.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T03:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T03:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T03:32:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T03:32:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T03:32:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T03:32:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 345 2024-04-25T03:32:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 346 2024-04-25T03:32:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T03:32:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 15ebe1b23927 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.15ebe1b23927 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:28 2024 2024-04-25T03:32:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:28 2024 2024-04-25T03:32:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:28 2024 2024-04-25T03:32:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 17db2559-bfc2-4c6d-a7f7-63f2838b82cd 2024-04-25T03:32:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T03:32:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T03:32:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T03:32:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: b250ce1f-2b98-476f-8e95-8fb5c7122260 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:29.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T03:32:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T03:32:29.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:29.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T03:32:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T03:32:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T03:32:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T03:32:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T03:32:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T03:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T03:32:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 348 2024-04-25T03:32:29.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 349 2024-04-25T03:32:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T03:32:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T03:32:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 15fd20106010 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.15fd20106010 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:29 2024 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:29 2024 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:29 2024 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:32:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T03:32:30.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T03:32:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T03:32:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: cadfb7b8-a23a-4bd3-88cb-6b1cda37b4fc 2024-04-25T03:32:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T03:32:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T03:32:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T03:32:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T03:32:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T03:32:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T03:32:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T03:32:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T03:32:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 352 2024-04-25T03:32:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 351 2024-04-25T03:32:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T03:32:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T03:32:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 160eec47bab4 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.160eec47bab4 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:30 2024 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:30 2024 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:30 2024 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:32:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T03:32:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T03:32:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T03:32:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: c823f1ec-09d5-471d-987d-719aa584a8bb 2024-04-25T03:32:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:31.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:31.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T03:32:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T03:32:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T03:32:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T03:32:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:32:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:32:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:32:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T03:32:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T03:32:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T03:32:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:32:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T03:32:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:32:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:32:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T03:32:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T03:32:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 354 2024-04-25T03:32:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:32:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:31.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 355 2024-04-25T03:32:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T03:32:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 356 2024-04-25T03:32:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1623171b108 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1623171b108 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:32:31 2024 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:32:31 2024 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:32:31 2024 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:32:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:32:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:32:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:32:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:32:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:32:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T03:32:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T03:32:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:32:32.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T03:32:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:32:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:32:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:32:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: e9883217-a73d-4a29-a67a-30a10f4a0db5 2024-04-25T03:32:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:32:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:32:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:32:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:32:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:32:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:32:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:32:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T03:32:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T03:32:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:32:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T03:32:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54593 2024-04-25T03:32:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T03:32:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T03:32:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T03:32:32.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:32:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T03:32:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 359 2024-04-25T03:32:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 358 2024-04-25T03:32:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 360 2024-04-25T03:32:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 362 2024-04-25T03:32:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 361 2024-04-25T03:32:33.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:32:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:32:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 364 2024-04-25T03:32:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 363 2024-04-25T03:32:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:32:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:32:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:32:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:32:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 365 2024-04-25T03:32:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 366 2024-04-25T03:32:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 367 2024-04-25T03:32:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 370 2024-04-25T03:32:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 369 2024-04-25T03:32:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 368 2024-04-25T03:32:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:32:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:32:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 371 2024-04-25T03:32:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 372 2024-04-25T03:32:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:32:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:32:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 373 2024-04-25T03:32:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 374 2024-04-25T03:32:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 375 2024-04-25T03:32:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 377 2024-04-25T03:32:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 376 2024-04-25T03:32:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:41.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 380 2024-04-25T03:32:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:41.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 378 2024-04-25T03:32:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 379 2024-04-25T03:32:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:32:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:32:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 381 2024-04-25T03:32:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 382 2024-04-25T03:32:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 384 2024-04-25T03:32:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 383 2024-04-25T03:32:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:32:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:32:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 386 2024-04-25T03:32:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 387 2024-04-25T03:32:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:32:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:32:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 388 2024-04-25T03:32:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 389 2024-04-25T03:32:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:45.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 385 2024-04-25T03:32:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:32:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:32:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 390 2024-04-25T03:32:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 391 2024-04-25T03:32:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:32:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:46.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:32:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 393 2024-04-25T03:32:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 392 2024-04-25T03:32:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:32:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 395 2024-04-25T03:32:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 394 2024-04-25T03:32:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 397 2024-04-25T03:32:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 396 2024-04-25T03:32:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 398 2024-04-25T03:32:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:32:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:32:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 399 2024-04-25T03:32:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 400 2024-04-25T03:32:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 401 2024-04-25T03:32:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:32:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 403 2024-04-25T03:32:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:32:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:32:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 402 2024-04-25T03:32:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 405 2024-04-25T03:32:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 404 2024-04-25T03:32:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 407 2024-04-25T03:32:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 406 2024-04-25T03:32:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:32:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:32:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:32:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 408 2024-04-25T03:32:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:32:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:32:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:32:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 409 2024-04-25T03:32:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:32:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 411 2024-04-25T03:32:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:32:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 410 2024-04-25T03:32:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:32:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:32:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:32:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:32:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 414 2024-04-25T03:32:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 412 2024-04-25T03:32:57.886 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:32:57.884+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966777c400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:32:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 415 2024-04-25T03:32:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:32:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:32:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:32:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 413 2024-04-25T03:32:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:32:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:32:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:32:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 417 2024-04-25T03:32:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:32:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:32:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:32:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 416 2024-04-25T03:32:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:59.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:32:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:32:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:32:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:32:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:32:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:32:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:32:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:32:59.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:32:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 418 2024-04-25T03:32:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 419 2024-04-25T03:32:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:32:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:00.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 421 2024-04-25T03:33:00.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 420 2024-04-25T03:33:00.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 423 2024-04-25T03:33:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 422 2024-04-25T03:33:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:02.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 425 2024-04-25T03:33:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:02.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:02.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 424 2024-04-25T03:33:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:03.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:03.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 427 2024-04-25T03:33:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 426 2024-04-25T03:33:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 429 2024-04-25T03:33:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 428 2024-04-25T03:33:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:06.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 430 2024-04-25T03:33:06.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 431 2024-04-25T03:33:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 434 2024-04-25T03:33:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 433 2024-04-25T03:33:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 432 2024-04-25T03:33:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 435 2024-04-25T03:33:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 436 2024-04-25T03:33:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 437 2024-04-25T03:33:08.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 439 2024-04-25T03:33:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 438 2024-04-25T03:33:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:10.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 442 2024-04-25T03:33:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 441 2024-04-25T03:33:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 440 2024-04-25T03:33:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:11.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:11.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 444 2024-04-25T03:33:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:12.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 443 2024-04-25T03:33:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 445 2024-04-25T03:33:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:12.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 447 2024-04-25T03:33:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 446 2024-04-25T03:33:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:13.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 448 2024-04-25T03:33:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 449 2024-04-25T03:33:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 451 2024-04-25T03:33:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 452 2024-04-25T03:33:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 450 2024-04-25T03:33:15.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:15.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 454 2024-04-25T03:33:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:16.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 453 2024-04-25T03:33:17.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 455 2024-04-25T03:33:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:17.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 456 2024-04-25T03:33:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 457 2024-04-25T03:33:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:18.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 459 2024-04-25T03:33:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 458 2024-04-25T03:33:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 460 2024-04-25T03:33:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:20.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 461 2024-04-25T03:33:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 462 2024-04-25T03:33:21.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 464 2024-04-25T03:33:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 463 2024-04-25T03:33:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:23.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 465 2024-04-25T03:33:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 467 2024-04-25T03:33:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:24.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 466 2024-04-25T03:33:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 468 2024-04-25T03:33:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 469 2024-04-25T03:33:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 472 2024-04-25T03:33:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 471 2024-04-25T03:33:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:26.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 474 2024-04-25T03:33:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 473 2024-04-25T03:33:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 475 2024-04-25T03:33:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 470 2024-04-25T03:33:29.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 476 2024-04-25T03:33:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 477 2024-04-25T03:33:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:30.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 479 2024-04-25T03:33:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 478 2024-04-25T03:33:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 480 2024-04-25T03:33:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 481 2024-04-25T03:33:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 483 2024-04-25T03:33:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 482 2024-04-25T03:33:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 485 2024-04-25T03:33:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 484 2024-04-25T03:33:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 487 2024-04-25T03:33:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 486 2024-04-25T03:33:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 488 2024-04-25T03:33:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 489 2024-04-25T03:33:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 490 2024-04-25T03:33:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 491 2024-04-25T03:33:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 493 2024-04-25T03:33:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 492 2024-04-25T03:33:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 494 2024-04-25T03:33:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 495 2024-04-25T03:33:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 496 2024-04-25T03:33:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 497 2024-04-25T03:33:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:38.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:39.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 499 2024-04-25T03:33:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 498 2024-04-25T03:33:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 500 2024-04-25T03:33:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:40.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 501 2024-04-25T03:33:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 503 2024-04-25T03:33:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 502 2024-04-25T03:33:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 505 2024-04-25T03:33:42.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 504 2024-04-25T03:33:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:43.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 507 2024-04-25T03:33:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 506 2024-04-25T03:33:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 509 2024-04-25T03:33:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 508 2024-04-25T03:33:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:44.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 511 2024-04-25T03:33:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 510 2024-04-25T03:33:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 513 2024-04-25T03:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 512 2024-04-25T03:33:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:48.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 514 2024-04-25T03:33:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 515 2024-04-25T03:33:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 517 2024-04-25T03:33:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 516 2024-04-25T03:33:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 518 2024-04-25T03:33:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 519 2024-04-25T03:33:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 520 2024-04-25T03:33:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 521 2024-04-25T03:33:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 523 2024-04-25T03:33:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:52.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 522 2024-04-25T03:33:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:52.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 524 2024-04-25T03:33:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 525 2024-04-25T03:33:52.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:53.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 526 2024-04-25T03:33:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 527 2024-04-25T03:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:33:54.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:33:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 529 2024-04-25T03:33:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 528 2024-04-25T03:33:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:33:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:33:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:33:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:33:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:33:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:33:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:33:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 530 2024-04-25T03:33:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 531 2024-04-25T03:33:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 533 2024-04-25T03:33:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 532 2024-04-25T03:33:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 535 2024-04-25T03:33:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 534 2024-04-25T03:33:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:33:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:33:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:33:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:33:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:33:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:33:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:33:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:33:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:33:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:33:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:33:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:33:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 536 2024-04-25T03:33:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 537 2024-04-25T03:33:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:33:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 538 2024-04-25T03:33:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:33:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:33:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:33:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:33:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:33:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 539 2024-04-25T03:34:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 541 2024-04-25T03:34:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 540 2024-04-25T03:34:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:02.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 543 2024-04-25T03:34:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 542 2024-04-25T03:34:02.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:03.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 544 2024-04-25T03:34:03.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 545 2024-04-25T03:34:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 547 2024-04-25T03:34:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 546 2024-04-25T03:34:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 549 2024-04-25T03:34:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 548 2024-04-25T03:34:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 550 2024-04-25T03:34:06.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 551 2024-04-25T03:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 553 2024-04-25T03:34:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:07.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 552 2024-04-25T03:34:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 555 2024-04-25T03:34:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 554 2024-04-25T03:34:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 556 2024-04-25T03:34:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 557 2024-04-25T03:34:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:09.959 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:34:09.956+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665735c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:34:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 558 2024-04-25T03:34:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 559 2024-04-25T03:34:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 560 2024-04-25T03:34:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 561 2024-04-25T03:34:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 563 2024-04-25T03:34:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 562 2024-04-25T03:34:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 565 2024-04-25T03:34:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 564 2024-04-25T03:34:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 567 2024-04-25T03:34:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 566 2024-04-25T03:34:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:13.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:14.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 569 2024-04-25T03:34:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 568 2024-04-25T03:34:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 570 2024-04-25T03:34:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 571 2024-04-25T03:34:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:15.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:15.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:16.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 573 2024-04-25T03:34:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 572 2024-04-25T03:34:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 574 2024-04-25T03:34:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:17.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 575 2024-04-25T03:34:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:18.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 577 2024-04-25T03:34:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 576 2024-04-25T03:34:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 578 2024-04-25T03:34:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 579 2024-04-25T03:34:20.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:20.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:34:20.989+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665a5cc00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:34:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 580 2024-04-25T03:34:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 581 2024-04-25T03:34:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:21.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 583 2024-04-25T03:34:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 582 2024-04-25T03:34:22.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 584 2024-04-25T03:34:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 585 2024-04-25T03:34:22.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 586 2024-04-25T03:34:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 587 2024-04-25T03:34:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 589 2024-04-25T03:34:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 588 2024-04-25T03:34:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 590 2024-04-25T03:34:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 591 2024-04-25T03:34:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 593 2024-04-25T03:34:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 592 2024-04-25T03:34:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 594 2024-04-25T03:34:27.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 595 2024-04-25T03:34:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:28.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 597 2024-04-25T03:34:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 596 2024-04-25T03:34:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:30.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 600 2024-04-25T03:34:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 599 2024-04-25T03:34:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 598 2024-04-25T03:34:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 601 2024-04-25T03:34:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 603 2024-04-25T03:34:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 602 2024-04-25T03:34:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 604 2024-04-25T03:34:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 605 2024-04-25T03:34:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 607 2024-04-25T03:34:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 608 2024-04-25T03:34:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 606 2024-04-25T03:34:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 610 2024-04-25T03:34:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:35.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 609 2024-04-25T03:34:36.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:34:36.019+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667960000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:34:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 611 2024-04-25T03:34:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 613 2024-04-25T03:34:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:37.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:37.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 612 2024-04-25T03:34:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:37.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:34:37.942+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665a5d000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:34:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 617 2024-04-25T03:34:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 614 2024-04-25T03:34:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 615 2024-04-25T03:34:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 616 2024-04-25T03:34:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 618 2024-04-25T03:34:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:39.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 619 2024-04-25T03:34:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 620 2024-04-25T03:34:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 621 2024-04-25T03:34:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:34:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:34:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:34:41.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 623 2024-04-25T03:34:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:41.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 622 2024-04-25T03:34:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 624 2024-04-25T03:34:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 626 2024-04-25T03:34:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:34:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:34:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:34:42.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 625 2024-04-25T03:34:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:43.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:34:43.937+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667530400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:34:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 628 2024-04-25T03:34:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 627 2024-04-25T03:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:44.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:34:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:34:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 630 2024-04-25T03:34:44.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 629 2024-04-25T03:34:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 631 2024-04-25T03:34:45.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:45.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:34:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:34:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:34:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 633 2024-04-25T03:34:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 634 2024-04-25T03:34:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 632 2024-04-25T03:34:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 635 2024-04-25T03:34:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:34:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:34:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:34:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54727 2024-04-25T03:34:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54861 2024-04-25T03:34:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:34:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:34:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:34:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:34:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:34:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:34:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 636 2024-04-25T03:34:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 637 2024-04-25T03:34:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:49.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:34:49.966+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667776c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:34:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 639 2024-04-25T03:34:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:34:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:34:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:34:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54995 2024-04-25T03:34:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55149 2024-04-25T03:34:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:34:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:34:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:34:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 638 2024-04-25T03:34:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55304 2024-04-25T03:34:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:34:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:34:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:34:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:34:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:34:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 640 2024-04-25T03:34:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55459 2024-04-25T03:34:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:53.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55614 2024-04-25T03:34:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 641 2024-04-25T03:34:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 642 2024-04-25T03:34:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:34:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55790 2024-04-25T03:34:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:34:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55966 2024-04-25T03:34:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:34:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:34:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54594 2024-04-25T03:37:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54728 2024-04-25T03:37:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54862 2024-04-25T03:37:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:25.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 54996 2024-04-25T03:37:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55150 2024-04-25T03:37:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55305 2024-04-25T03:37:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55460 2024-04-25T03:37:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55615 2024-04-25T03:37:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55791 2024-04-25T03:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:37:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 55967 2024-04-25T03:37:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:37:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:37:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:37:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T03:37:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T03:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T03:37:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T03:37:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T03:37:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T03:37:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T03:37:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T03:37:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T03:37:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T03:37:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T03:37:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61959 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T03:37:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T03:37:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T03:37:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T03:37:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T03:37:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T03:37:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T03:37:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T03:37:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:37:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T03:37:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:37:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:37:33.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:37:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:37:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:33.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:37:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:37:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:37:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:37:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:34.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:37:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:37:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:38:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=50da92d2639477a3c0aa17044be378a3 2024-04-25T03:38:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T03:38:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=a275bb2614ccadc022f661bf09b5f698 2024-04-25T03:38:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T03:38:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=652ae6544c9bc6fc93ab564959e22ce1 2024-04-25T03:38:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T03:38:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=7a7ffd2f919d8f51072dc4211a782267 2024-04-25T03:38:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T03:38:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T03:38:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T03:38:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T03:38:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T03:38:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T03:38:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T03:38:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=5bad78fde84a29b6e38d2c19b88f5417 2024-04-25T03:38:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T03:38:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T03:38:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T03:38:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=8f0736d90b9f90a1d9d463aac96c0fca 2024-04-25T03:38:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T03:38:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T03:38:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T03:38:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T03:38:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T03:38:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=3823d3ee0df0834a7b1ca748d9cd66fd 2024-04-25T03:38:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T03:38:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T03:38:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T03:38:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T03:38:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=31c00d4fe7cfa7515cc2625d448500f4 2024-04-25T03:38:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T03:38:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T03:38:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T03:38:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=8c5c6c98dd5a0afec4d4976f64e49424 2024-04-25T03:38:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T03:38:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T03:38:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T03:38:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=03a45430db2dd9a960ad7e4bbb091430 2024-04-25T03:38:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T03:38:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T03:38:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T03:38:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:38:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:38:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:38:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:38:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:38:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:38:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:38:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:38:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:38:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:38:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:38:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:38:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:38:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:38:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:38:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:38:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:38:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:38:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:38:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:38:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:38:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:38:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:38:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e3bb2dd3-8ca0-4a47-8072-65e4a79f4950 2024-04-25T03:38:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:32.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T03:38:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:33.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:38:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:38:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:38:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:38:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:38:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:38:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:38:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:38:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:33.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:33.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:33.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:38:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:38:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:33.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:38:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17db2559-bfc2-4c6d-a7f7-63f2838b82cd 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eb5ea9d4-e7be-4bfc-8787-d1c9443393ff 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:38:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:38:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:38:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:38:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:34.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:38:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:38:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:38:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e3bb2dd3-8ca0-4a47-8072-65e4a79f4950 2024-04-25T03:38:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:35.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:38:35.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:38:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:38:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:38:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:36.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:38:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:38:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4 2024-04-25T03:38:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:38:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:38:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17db2559-bfc2-4c6d-a7f7-63f2838b82cd 2024-04-25T03:38:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:38:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eb5ea9d4-e7be-4bfc-8787-d1c9443393ff 2024-04-25T03:38:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:36.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:38:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:37.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:38:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:38:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:38:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:38:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:38:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:38:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:38:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:38:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:38:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:38:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:38:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:38:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:38:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:38:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:38:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:38:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:38:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4 2024-04-25T03:38:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:38:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:38:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:38:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:38:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:38:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e3bb2dd3-8ca0-4a47-8072-65e4a79f4950 2024-04-25T03:38:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:38:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:38:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:38:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17db2559-bfc2-4c6d-a7f7-63f2838b82cd 2024-04-25T03:38:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:38:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eb5ea9d4-e7be-4bfc-8787-d1c9443393ff 2024-04-25T03:38:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:40.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:38:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:38:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:38:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4 2024-04-25T03:38:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:38:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:38:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:38:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:38:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:38:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:38:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:38:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:38:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:38:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:38:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:45.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:38:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:38:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:38:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:38:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:45.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:38:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:38:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:45.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:38:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:38:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:38:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:38:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:38:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:38:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:38:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:38:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:16' 2024-04-25T03:38:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:38:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:38:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:38:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:38:46.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:38:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e3bb2dd3-8ca0-4a47-8072-65e4a79f4950 2024-04-25T03:38:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:38:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:38:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T03:38:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:38:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e3bb2dd3-8ca0-4a47-8072-65e4a79f4950 2024-04-25T03:38:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:48.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:38:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:38:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:38:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:38:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T03:38:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:38:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T03:38:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:38:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eb5ea9d4-e7be-4bfc-8787-d1c9443393ff 2024-04-25T03:38:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:38:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17db2559-bfc2-4c6d-a7f7-63f2838b82cd 2024-04-25T03:38:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:48.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:38:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:38:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:38:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:38:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T03:38:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T03:38:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:38:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T03:38:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eb5ea9d4-e7be-4bfc-8787-d1c9443393ff 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:38:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17db2559-bfc2-4c6d-a7f7-63f2838b82cd 2024-04-25T03:38:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:38:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:38:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:38:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:38:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:49.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T03:38:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T03:38:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:38:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:38:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:38:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4 2024-04-25T03:38:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:38:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:38:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T03:38:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T03:38:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T03:38:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T03:38:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:38:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:38:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T03:38:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:38:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4 2024-04-25T03:38:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:51.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:38:51.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:38:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:51.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T03:38:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:38:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T03:38:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:52.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T03:38:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T03:38:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:38:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:38:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:38:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:38:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:38:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:38:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:38:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:38:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:38:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:53.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:38:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:38:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:38:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:54.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:38:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:38:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:38:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:38:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T03:38:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:38:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T03:38:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:38:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:38:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:54.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T03:38:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:38:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:38:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:54.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:38:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:38:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:38:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:38:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:38:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:38:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:54.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:54.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:38:54.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:38:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T03:38:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T03:38:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T03:38:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T03:38:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T03:38:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T03:38:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T03:38:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T03:38:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:38:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:38:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:38:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:38:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:38:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:38:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:55.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:38:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:38:55.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T03:38:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:38:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T03:38:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:38:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:38:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:38:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:38:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:38:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:38:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T03:38:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:38:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T03:38:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T03:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:38:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T03:38:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T03:38:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:38:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:38:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T03:38:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:38:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T03:38:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T03:38:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:38:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:38:55.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T03:38:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:38:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:38:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:38:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:38:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:38:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:38:46' 2024-04-25T03:38:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:38:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:38:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:38:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:38:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:38:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:38:56.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:38:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:38:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:38:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 03:38:46' 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:38:46' 2024-04-25T03:38:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:38:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:38:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:38:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:38:46' 2024-04-25T03:38:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:38:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:38:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:38:46' 2024-04-25T03:38:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:38:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:38:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:38:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:38:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:38:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T03:38:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:38:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T03:38:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T03:38:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T03:38:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T03:38:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T03:38:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:38:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:38:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T03:38:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T03:38:57.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:38:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T03:38:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:38:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T03:38:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:38:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T03:38:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:38:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T03:38:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:38:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:38:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T03:38:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:38:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:39:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=a275bb2614ccadc022f661bf09b5f698 2024-04-25T03:39:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T03:39:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ a275bb2614ccadc022f661bf09b5f698 != \a\2\7\5\b\b\2\6\1\4\c\c\a\d\c\0\2\2\f\6\6\1\b\f\0\9\b\5\f\6\9\8 ]] 2024-04-25T03:39:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61960 2024-04-25T03:39:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=03a45430db2dd9a960ad7e4bbb091430 2024-04-25T03:39:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T03:39:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 03a45430db2dd9a960ad7e4bbb091430 != \0\3\a\4\5\4\3\0\d\b\2\d\d\9\a\9\6\0\a\d\7\e\4\b\b\b\0\9\1\4\3\0 ]] 2024-04-25T03:39:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=8c5c6c98dd5a0afec4d4976f64e49424 2024-04-25T03:39:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T03:39:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 8c5c6c98dd5a0afec4d4976f64e49424 != \8\c\5\c\6\c\9\8\d\d\5\a\0\a\f\e\c\4\d\4\9\7\6\f\6\4\e\4\9\4\2\4 ]] 2024-04-25T03:39:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=31c00d4fe7cfa7515cc2625d448500f4 2024-04-25T03:39:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T03:39:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 31c00d4fe7cfa7515cc2625d448500f4 != \3\1\c\0\0\d\4\f\e\7\c\f\a\7\5\1\5\c\c\2\6\2\5\d\4\4\8\5\0\0\f\4 ]] 2024-04-25T03:39:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61962 2024-04-25T03:39:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=8f0736d90b9f90a1d9d463aac96c0fca 2024-04-25T03:39:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T03:39:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=5bad78fde84a29b6e38d2c19b88f5417 2024-04-25T03:39:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T03:39:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 8f0736d90b9f90a1d9d463aac96c0fca != \8\f\0\7\3\6\d\9\0\b\9\f\9\0\a\1\d\9\d\4\6\3\a\a\c\9\6\c\0\f\c\a ]] 2024-04-25T03:39:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 5bad78fde84a29b6e38d2c19b88f5417 != \5\b\a\d\7\8\f\d\e\8\4\a\2\9\b\6\e\3\8\d\2\c\1\9\b\8\8\f\5\4\1\7 ]] 2024-04-25T03:39:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61964 2024-04-25T03:39:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61966 2024-04-25T03:39:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=3823d3ee0df0834a7b1ca748d9cd66fd 2024-04-25T03:39:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T03:39:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 3823d3ee0df0834a7b1ca748d9cd66fd != \3\8\2\3\d\3\e\e\0\d\f\0\8\3\4\a\7\b\1\c\a\7\4\8\d\9\c\d\6\6\f\d ]] 2024-04-25T03:39:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61968 2024-04-25T03:39:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=652ae6544c9bc6fc93ab564959e22ce1 2024-04-25T03:39:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T03:39:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=50da92d2639477a3c0aa17044be378a3 2024-04-25T03:39:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T03:39:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 652ae6544c9bc6fc93ab564959e22ce1 != \6\5\2\a\e\6\5\4\4\c\9\b\c\6\f\c\9\3\a\b\5\6\4\9\5\9\e\2\2\c\e\1 ]] 2024-04-25T03:39:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 50da92d2639477a3c0aa17044be378a3 != \5\0\d\a\9\2\d\2\6\3\9\4\7\7\a\3\c\0\a\a\1\7\0\4\4\b\e\3\7\8\a\3 ]] 2024-04-25T03:39:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61970 2024-04-25T03:39:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:51.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61972 2024-04-25T03:39:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=7a7ffd2f919d8f51072dc4211a782267 2024-04-25T03:39:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T03:39:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 7a7ffd2f919d8f51072dc4211a782267 != \7\a\7\f\f\d\2\f\9\1\9\d\8\f\5\1\0\7\2\d\c\4\2\1\1\a\7\8\2\2\6\7 ]] 2024-04-25T03:39:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61974 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 61975 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:39:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:39:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:39:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:39:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:39:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:39:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:39:53.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e3bb2dd3-8ca0-4a47-8072-65e4a79f4950 2024-04-25T03:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3636633.6,"bytes_per_snapshot":72732672.0,"last_snapshot_bytes":72732672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016328,"remote_snapshot_timestamp":1714016328,"replay_state":"idle"} 2024-04-25T03:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:39:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:39:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:39:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:39:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3636633.6,"bytes_per_snapshot":72732672.0,"last_snapshot_bytes":72732672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016328,"remote_snapshot_timestamp":1714016328,"replay_state":"idle"} 2024-04-25T03:39:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:39:53.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:39:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:39:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:39:53.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:39:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:39:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:39:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:39:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:39:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:39:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:39:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1e3bb2dd3-8ca0-4a47-8072-65e4a79f4950up+replayingreplaying, {"bytes_per_second":3636633.6,"bytes_per_snapshot":72732672.0,"last_snapshot_bytes":72732672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016328,"remote_snapshot_timestamp":1714016328,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+replayingreplaying, {"bytes_per_second":3689267.2,"bytes_per_snapshot":73785344.0,"last_snapshot_bytes":73785344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016332,"remote_snapshot_timestamp":1714016332,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+replayingreplaying, {"bytes_per_second":3174400.0,"bytes_per_snapshot":63488000.0,"last_snapshot_bytes":63488000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+replayingreplaying, {"bytes_per_second":3682918.4,"bytes_per_snapshot":73658368.0,"last_snapshot_bytes":73658368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+replayingreplaying, {"bytes_per_second":3620454.4,"bytes_per_snapshot":72409088.0,"last_snapshot_bytes":72409088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+replayingreplaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+replayingreplaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:39:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:39:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:39:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:39:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:39:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:39:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:39:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:39:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:39:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T03:39:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:39:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:39:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T03:39:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T03:39:53.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:39:53.778+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664480b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:39:53.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:39:53.778+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/e3bb2dd3-8ca0-4a47-8072-65e4a79f4950] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:39:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:39:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T03:39:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:39:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:39:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:39:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T03:39:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T03:39:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:39:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:39:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:39:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:39:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:39:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:39:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:39:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4 2024-04-25T03:39:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:39:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3689267.2,"bytes_per_snapshot":73785344.0,"last_snapshot_bytes":73785344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016332,"remote_snapshot_timestamp":1714016332,"replay_state":"idle"} 2024-04-25T03:39:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:39:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:39:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:39:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:39:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:39:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3689267.2,"bytes_per_snapshot":73785344.0,"last_snapshot_bytes":73785344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016332,"remote_snapshot_timestamp":1714016332,"replay_state":"idle"} 2024-04-25T03:39:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:39:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:39:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:39:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:39:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:39:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:39:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:39:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:39:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:39:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:39:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:39:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary2da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4up+replayingreplaying, {"bytes_per_second":3689267.2,"bytes_per_snapshot":73785344.0,"last_snapshot_bytes":73785344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016332,"remote_snapshot_timestamp":1714016332,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+replayingreplaying, {"bytes_per_second":3174400.0,"bytes_per_snapshot":63488000.0,"last_snapshot_bytes":63488000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+replayingreplaying, {"bytes_per_second":3682918.4,"bytes_per_snapshot":73658368.0,"last_snapshot_bytes":73658368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+replayingreplaying, {"bytes_per_second":3620454.4,"bytes_per_snapshot":72409088.0,"last_snapshot_bytes":72409088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+replayingreplaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+replayingreplaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:39:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:39:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:39:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:39:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:39:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:39:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:39:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:39:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:39:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T03:39:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:39:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:39:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T03:39:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T03:39:56.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:39:56.634+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667703d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:39:56.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:39:56.634+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/da94d8c3-93b4-41a8-9ca2-6dc6b11d54a4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:39:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:39:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T03:39:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:39:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:39:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:39:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T03:39:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T03:39:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:39:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:39:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T03:39:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:39:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:39:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:39:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:39:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:39:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:39:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aeb686a6-8fb9-4f61-9cc8-5c0179cf433e 2024-04-25T03:39:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:39:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3174400.0,"bytes_per_snapshot":63488000.0,"last_snapshot_bytes":63488000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:39:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:39:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:39:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:39:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:39:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:39:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3174400.0,"bytes_per_snapshot":63488000.0,"last_snapshot_bytes":63488000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:39:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:39:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:39:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:39:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:39:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:39:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:39:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:39:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:39:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:39:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:39:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:39:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:39:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:39:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:39:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:39:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:39:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:39:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary3aeb686a6-8fb9-4f61-9cc8-5c0179cf433eup+replayingreplaying, {"bytes_per_second":3174400.0,"bytes_per_snapshot":63488000.0,"last_snapshot_bytes":63488000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+replayingreplaying, {"bytes_per_second":3682918.4,"bytes_per_snapshot":73658368.0,"last_snapshot_bytes":73658368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+replayingreplaying, {"bytes_per_second":3620454.4,"bytes_per_snapshot":72409088.0,"last_snapshot_bytes":72409088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+replayingreplaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+replayingreplaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:39:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:39:59.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:39:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:39:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:39:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:39:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:39:59.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:39:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:39:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:39:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T03:39:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:39:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:39:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:39:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T03:39:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T03:39:59.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:39:59.630+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667702820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:39:59.633 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:39:59.630+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/aeb686a6-8fb9-4f61-9cc8-5c0179cf433e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T03:40:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T03:40:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:40:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:40:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eb5ea9d4-e7be-4bfc-8787-d1c9443393ff 2024-04-25T03:40:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:40:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3682918.4,"bytes_per_snapshot":73658368.0,"last_snapshot_bytes":73658368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"} 2024-04-25T03:40:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:40:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:40:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:40:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:40:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:40:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3682918.4,"bytes_per_snapshot":73658368.0,"last_snapshot_bytes":73658368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"} 2024-04-25T03:40:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:40:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:40:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:40:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:40:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:40:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:40:02.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:40:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary4eb5ea9d4-e7be-4bfc-8787-d1c9443393ffup+replayingreplaying, {"bytes_per_second":3682918.4,"bytes_per_snapshot":73658368.0,"last_snapshot_bytes":73658368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+replayingreplaying, {"bytes_per_second":3620454.4,"bytes_per_snapshot":72409088.0,"last_snapshot_bytes":72409088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+replayingreplaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+replayingreplaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:40:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:40:02.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:40:02.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:40:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:40:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:40:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T03:40:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:40:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T03:40:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T03:40:02.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:02.653+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c5c820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:40:02.656 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:02.653+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb680 [3/eb5ea9d4-e7be-4bfc-8787-d1c9443393ff] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T03:40:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T03:40:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T03:40:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:40:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T03:40:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:40:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:40:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 89ea54cb-4ea2-4469-9410-cfe782ebe41c 2024-04-25T03:40:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:40:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3620454.4,"bytes_per_snapshot":72409088.0,"last_snapshot_bytes":72409088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:40:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:40:05.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:40:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:40:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:40:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3620454.4,"bytes_per_snapshot":72409088.0,"last_snapshot_bytes":72409088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:40:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:40:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:40:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:40:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:40:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:40:05.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:40:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:40:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:40:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary589ea54cb-4ea2-4469-9410-cfe782ebe41cup+replayingreplaying, {"bytes_per_second":3620454.4,"bytes_per_snapshot":72409088.0,"last_snapshot_bytes":72409088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+replayingreplaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+replayingreplaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:40:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:40:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:40:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:40:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:40:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:40:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T03:40:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:40:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T03:40:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T03:40:05.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:05.673+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667702b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:40:05.676 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:05.673+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/89ea54cb-4ea2-4469-9410-cfe782ebe41c] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T03:40:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T03:40:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T03:40:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:40:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T03:40:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:07.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:40:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:40:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:40:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 18b2adf1-dfec-4df8-8956-32fa3603bb6f 2024-04-25T03:40:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:40:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:40:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:40:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:40:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:40:08.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:40:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:40:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:40:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:08.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:40:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:40:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:40:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:40:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary618b2adf1-dfec-4df8-8956-32fa3603bb6fup+replayingreplaying, {"bytes_per_second":3221504.0,"bytes_per_snapshot":64430080.0,"last_snapshot_bytes":64430080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+replayingreplaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:40:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:40:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:08.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:40:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:40:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:40:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T03:40:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:40:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T03:40:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T03:40:08.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:08.594+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664a6a9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:40:08.598 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:08.594+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdaa00 [3/18b2adf1-dfec-4df8-8956-32fa3603bb6f] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T03:40:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T03:40:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T03:40:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:40:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T03:40:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:40:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:40:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17db2559-bfc2-4c6d-a7f7-63f2838b82cd 2024-04-25T03:40:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:40:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"} 2024-04-25T03:40:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:40:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:40:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:40:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:40:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:40:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"} 2024-04-25T03:40:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:40:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:40:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:40:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:40:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:40:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:40:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:40:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:40:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:40:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary717db2559-bfc2-4c6d-a7f7-63f2838b82cdup+replayingreplaying, {"bytes_per_second":3693977.6,"bytes_per_snapshot":73879552.0,"last_snapshot_bytes":73879552,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016330,"remote_snapshot_timestamp":1714016330,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:40:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:40:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:40:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:40:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:40:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:40:11.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T03:40:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:40:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T03:40:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T03:40:11.659 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:11.656+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c63ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:40:11.660 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:11.656+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/17db2559-bfc2-4c6d-a7f7-63f2838b82cd] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:13.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T03:40:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T03:40:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:40:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T03:40:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:14.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:40:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:40:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 757e7933-da6b-4849-9fe8-8f009ae4f952 2024-04-25T03:40:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:40:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:40:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:39:46' 2024-04-25T03:40:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:40:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:40:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:40:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:40:14.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:40:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:40:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:40:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:40:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 03:39:46' 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:39:46' 2024-04-25T03:40:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:40:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":3618816.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary8757e7933-da6b-4849-9fe8-8f009ae4f952up+replayingreplaying, {"bytes_per_second":3578880.0,"bytes_per_snapshot":71577600.0,"last_snapshot_bytes":71577600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":3614310.4,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:39:46' 2024-04-25T03:40:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:40:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:40:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:39:46' 2024-04-25T03:40:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:40:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:40:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:40:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T03:40:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:40:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T03:40:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T03:40:14.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:14.461+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596652f6000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:40:14.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:14.461+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/757e7933-da6b-4849-9fe8-8f009ae4f952] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T03:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T03:40:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T03:40:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:40:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T03:40:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 590aa47c-e611-4802-aae5-eb4e8c112ad6 2024-04-25T03:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:40:16' 2024-04-25T03:40:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:40:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:40:17.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:17.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:40:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:40:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:40:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:40:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:40:16' 2024-04-25T03:40:17.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 03:40:16' 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:40:16' 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:17.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:40:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:40:16image-primary9590aa47c-e611-4802-aae5-eb4e8c112ad6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72286208.0,"last_snapshot_bytes":72286208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:40:16' 2024-04-25T03:40:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:40:17.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:40:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:40:16' 2024-04-25T03:40:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:40:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:40:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:40:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T03:40:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:40:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T03:40:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T03:40:17.571 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:17.568+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667a4e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:40:17.572 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:17.568+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/590aa47c-e611-4802-aae5-eb4e8c112ad6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T03:40:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T03:40:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T03:40:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:40:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed 2024-04-25T03:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:40:16' 2024-04-25T03:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:40:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:40:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"} 2024-04-25T03:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:40:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:40:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:40:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:40:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:40:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:40:16' 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 03:40:16' 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:40:16' 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:40:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:40:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:40:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10667f478d-f9f7-4f0b-8f3d-1d3d9a6c86edup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72376320.0,"last_snapshot_bytes":72376320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016336,"remote_snapshot_timestamp":1714016336,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:40:16' 2024-04-25T03:40:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:40:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:40:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:40:16' 2024-04-25T03:40:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:40:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:40:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:40:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:40:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T03:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T03:40:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T03:40:20.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:20.505+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596652f61a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:40:20.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:20.505+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdac80 [3/667f478d-f9f7-4f0b-8f3d-1d3d9a6c86ed] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:40:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:40:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T03:40:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:22.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:40:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:40:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T03:40:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T03:40:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T03:40:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T03:40:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T03:40:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T03:40:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T03:40:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T03:40:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T03:40:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T03:40:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T03:40:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T03:40:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T03:40:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1a8859a478e8 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1a8859a478e8 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:22 2024 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:22 2024 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:22 2024 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ce9a2183-b3fb-4968-905c-afc3395a175e 2024-04-25T03:40:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T03:40:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T03:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T03:40:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 4e6c7ff2-6224-49b6-bd2f-bf7aa902712a 2024-04-25T03:40:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T03:40:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T03:40:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T03:40:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:40:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T03:40:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T03:40:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T03:40:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 664 2024-04-25T03:40:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:23.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T03:40:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1a9770f7f350 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1a9770f7f350 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:23 2024 2024-04-25T03:40:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:23 2024 2024-04-25T03:40:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:23 2024 2024-04-25T03:40:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: e77e2d29-1e62-423a-ac2a-2802074ee891 2024-04-25T03:40:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T03:40:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T03:40:24.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T03:40:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 95dfd8c8-df6e-49d2-acc8-c28771fb0b80 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T03:40:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T03:40:24.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T03:40:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T03:40:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T03:40:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T03:40:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 666 2024-04-25T03:40:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T03:40:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T03:40:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1aa5fb0b0172 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1aa5fb0b0172 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:24 2024 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:24 2024 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:24 2024 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 9dc2fa16-a672-4b8a-bd96-3f12debbec37 2024-04-25T03:40:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:25.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T03:40:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T03:40:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T03:40:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 11e34270-4ae0-4a8a-a217-fbf8433a4581 2024-04-25T03:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T03:40:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T03:40:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T03:40:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T03:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:25.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T03:40:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T03:40:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 668 2024-04-25T03:40:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T03:40:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T03:40:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1ab3bb14b14f 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1ab3bb14b14f 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:25 2024 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:25 2024 2024-04-25T03:40:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:25 2024 2024-04-25T03:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 72053786-3993-4b5c-8420-674034616008 2024-04-25T03:40:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T03:40:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T03:40:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T03:40:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 8859af56-cc69-489e-a70d-0df43a359a56 2024-04-25T03:40:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:26.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T03:40:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T03:40:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T03:40:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T03:40:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:40:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T03:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:26.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T03:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T03:40:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 670 2024-04-25T03:40:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T03:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T03:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1ac2ac4db93e 2024-04-25T03:40:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1ac2ac4db93e 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:26 2024 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:26 2024 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:26 2024 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ae95fc20-1bd3-40e6-a713-6cef123e8a98 2024-04-25T03:40:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:27.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T03:40:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T03:40:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T03:40:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 94856e45-6bee-4847-8f72-9dc148e5cc85 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:27.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:27.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T03:40:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T03:40:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T03:40:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T03:40:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T03:40:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T03:40:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T03:40:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 673 2024-04-25T03:40:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 672 2024-04-25T03:40:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T03:40:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T03:40:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1ad48a5a316d 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1ad48a5a316d 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:27 2024 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:27 2024 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:27 2024 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 1d680229-5269-461c-bcc8-835d37bcb581 2024-04-25T03:40:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T03:40:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T03:40:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T03:40:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 1ad190d8-1b25-481e-8e11-ee854d4ef8dd 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:28.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T03:40:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T03:40:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T03:40:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T03:40:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T03:40:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T03:40:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:29.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 676 2024-04-25T03:40:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 675 2024-04-25T03:40:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T03:40:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1ae6de652ca0 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1ae6de652ca0 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:28 2024 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:28 2024 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:28 2024 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 634e8d7a-89d2-4c21-afd8-664d6872f4ef 2024-04-25T03:40:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T03:40:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T03:40:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T03:40:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 11c9a95a-805a-40cd-96c8-22d044f3327f 2024-04-25T03:40:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T03:40:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T03:40:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T03:40:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T03:40:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T03:40:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 679 2024-04-25T03:40:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 678 2024-04-25T03:40:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T03:40:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1af7fd476f6f 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1af7fd476f6f 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:29 2024 2024-04-25T03:40:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:29 2024 2024-04-25T03:40:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:29 2024 2024-04-25T03:40:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 17eed5eb-d5ab-413a-be73-f100e37f507c 2024-04-25T03:40:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T03:40:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T03:40:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T03:40:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 58eecca8-5c1c-4b8b-8ad3-2f3684b26e11 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:30.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:30.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T03:40:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T03:40:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T03:40:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T03:40:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T03:40:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 681 2024-04-25T03:40:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 682 2024-04-25T03:40:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T03:40:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T03:40:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1b095fbb19de 2024-04-25T03:40:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1b095fbb19de 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:30 2024 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:30 2024 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:30 2024 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: f4cd445f-0346-4445-abe3-a66a688d8165 2024-04-25T03:40:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T03:40:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T03:40:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T03:40:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 6138ea12-1ea4-4266-b036-4b021a0edabb 2024-04-25T03:40:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T03:40:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T03:40:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:31.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T03:40:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:40:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T03:40:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T03:40:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:40:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:40:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T03:40:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T03:40:32.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 684 2024-04-25T03:40:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:40:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:32.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T03:40:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 685 2024-04-25T03:40:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 686 2024-04-25T03:40:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T03:40:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:40:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:40:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:40:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1b1e11cf3acd 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1b1e11cf3acd 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:40:31 2024 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:40:31 2024 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:40:31 2024 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 4baef160-bd07-4521-ad9d-62c56fa53b00 2024-04-25T03:40:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:40:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:40:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:40:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T03:40:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T03:40:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:40:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T03:40:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:40:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:40:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:40:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: c429965a-1d4d-4790-afd8-7bb80772d00f 2024-04-25T03:40:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:40:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:40:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:40:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:40:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:40:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:40:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:40:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T03:40:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T03:40:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:40:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T03:40:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:40:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66436 2024-04-25T03:40:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T03:40:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T03:40:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T03:40:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 690 2024-04-25T03:40:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 688 2024-04-25T03:40:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:40:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T03:40:33.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 689 2024-04-25T03:40:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:40:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:40:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:40:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:40:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 692 2024-04-25T03:40:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 691 2024-04-25T03:40:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 693 2024-04-25T03:40:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 694 2024-04-25T03:40:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:40:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:36.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 696 2024-04-25T03:40:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:36.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 695 2024-04-25T03:40:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 697 2024-04-25T03:40:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:40:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 699 2024-04-25T03:40:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 700 2024-04-25T03:40:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 698 2024-04-25T03:40:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 701 2024-04-25T03:40:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 702 2024-04-25T03:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:40:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:40:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 704 2024-04-25T03:40:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 703 2024-04-25T03:40:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:39.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:40:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:40:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:40.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 706 2024-04-25T03:40:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 705 2024-04-25T03:40:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 707 2024-04-25T03:40:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:40.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:40.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:40:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:40:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:40:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:40:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 709 2024-04-25T03:40:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 710 2024-04-25T03:40:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 708 2024-04-25T03:40:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:40:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:40:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 711 2024-04-25T03:40:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 712 2024-04-25T03:40:42.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 714 2024-04-25T03:40:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 713 2024-04-25T03:40:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:40:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:40:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:43.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 715 2024-04-25T03:40:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 716 2024-04-25T03:40:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:40:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:40:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 717 2024-04-25T03:40:45.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 718 2024-04-25T03:40:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 720 2024-04-25T03:40:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:40:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:45.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:40:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:40:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:46.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 719 2024-04-25T03:40:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 721 2024-04-25T03:40:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:40:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:40:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 724 2024-04-25T03:40:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 722 2024-04-25T03:40:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 725 2024-04-25T03:40:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:40:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:40:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 723 2024-04-25T03:40:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 728 2024-04-25T03:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:40:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:40:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 726 2024-04-25T03:40:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:50.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:50.020+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596677d3000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:40:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:40:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:40:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 729 2024-04-25T03:40:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 727 2024-04-25T03:40:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 731 2024-04-25T03:40:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 730 2024-04-25T03:40:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:40:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:40:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 733 2024-04-25T03:40:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:40:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:40:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 732 2024-04-25T03:40:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 734 2024-04-25T03:40:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:53.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 735 2024-04-25T03:40:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:53.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 736 2024-04-25T03:40:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 737 2024-04-25T03:40:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:40:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:40:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:40:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 739 2024-04-25T03:40:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 738 2024-04-25T03:40:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:40:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:40:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:40:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:40:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:40:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:40:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 741 2024-04-25T03:40:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:40:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:40:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:40:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:40:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:40:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:40:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 740 2024-04-25T03:40:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:40:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:40:57.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:40:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:40:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:40:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 743 2024-04-25T03:40:57.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:57.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 742 2024-04-25T03:40:58.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:40:58.034+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665734800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:40:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:40:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:40:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:40:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:40:58.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:40:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:40:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 744 2024-04-25T03:40:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 745 2024-04-25T03:40:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 746 2024-04-25T03:40:59.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 747 2024-04-25T03:40:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:40:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:40:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:40:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:40:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:40:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:40:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 748 2024-04-25T03:40:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:40:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 749 2024-04-25T03:41:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 750 2024-04-25T03:41:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:01.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 751 2024-04-25T03:41:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 753 2024-04-25T03:41:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:02.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 752 2024-04-25T03:41:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 754 2024-04-25T03:41:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 755 2024-04-25T03:41:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 758 2024-04-25T03:41:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:03.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:03.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:04.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 756 2024-04-25T03:41:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 757 2024-04-25T03:41:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:05.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:06.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 759 2024-04-25T03:41:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 760 2024-04-25T03:41:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 761 2024-04-25T03:41:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 766 2024-04-25T03:41:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 764 2024-04-25T03:41:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 763 2024-04-25T03:41:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 762 2024-04-25T03:41:08.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 765 2024-04-25T03:41:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:09.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 767 2024-04-25T03:41:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 768 2024-04-25T03:41:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:09.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 769 2024-04-25T03:41:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:10.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 770 2024-04-25T03:41:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 772 2024-04-25T03:41:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 771 2024-04-25T03:41:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 774 2024-04-25T03:41:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:12.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 775 2024-04-25T03:41:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 773 2024-04-25T03:41:13.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 776 2024-04-25T03:41:13.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:14.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:41:14.097+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664e57c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 777 2024-04-25T03:41:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 778 2024-04-25T03:41:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:15.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:15.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 782 2024-04-25T03:41:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 779 2024-04-25T03:41:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 781 2024-04-25T03:41:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 780 2024-04-25T03:41:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 783 2024-04-25T03:41:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:17.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:17.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 785 2024-04-25T03:41:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 784 2024-04-25T03:41:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 786 2024-04-25T03:41:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 788 2024-04-25T03:41:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 787 2024-04-25T03:41:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 790 2024-04-25T03:41:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 789 2024-04-25T03:41:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 792 2024-04-25T03:41:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 791 2024-04-25T03:41:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:22.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:22.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 794 2024-04-25T03:41:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 795 2024-04-25T03:41:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 796 2024-04-25T03:41:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 793 2024-04-25T03:41:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 797 2024-04-25T03:41:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 798 2024-04-25T03:41:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 799 2024-04-25T03:41:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 802 2024-04-25T03:41:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 801 2024-04-25T03:41:26.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 803 2024-04-25T03:41:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 804 2024-04-25T03:41:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 800 2024-04-25T03:41:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 805 2024-04-25T03:41:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 807 2024-04-25T03:41:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 808 2024-04-25T03:41:29.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 809 2024-04-25T03:41:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 806 2024-04-25T03:41:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 810 2024-04-25T03:41:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 811 2024-04-25T03:41:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 812 2024-04-25T03:41:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 813 2024-04-25T03:41:33.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 814 2024-04-25T03:41:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 815 2024-04-25T03:41:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 817 2024-04-25T03:41:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 816 2024-04-25T03:41:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:34.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 818 2024-04-25T03:41:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 819 2024-04-25T03:41:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 820 2024-04-25T03:41:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 821 2024-04-25T03:41:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 823 2024-04-25T03:41:37.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 822 2024-04-25T03:41:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:37.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 825 2024-04-25T03:41:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 824 2024-04-25T03:41:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 826 2024-04-25T03:41:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 827 2024-04-25T03:41:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 829 2024-04-25T03:41:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 828 2024-04-25T03:41:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 831 2024-04-25T03:41:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 830 2024-04-25T03:41:41.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 833 2024-04-25T03:41:42.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 832 2024-04-25T03:41:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 835 2024-04-25T03:41:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 834 2024-04-25T03:41:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:44.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:44.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 836 2024-04-25T03:41:45.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 837 2024-04-25T03:41:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:46.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 838 2024-04-25T03:41:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 839 2024-04-25T03:41:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 841 2024-04-25T03:41:47.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 840 2024-04-25T03:41:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 842 2024-04-25T03:41:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 843 2024-04-25T03:41:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 845 2024-04-25T03:41:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 844 2024-04-25T03:41:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 846 2024-04-25T03:41:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 847 2024-04-25T03:41:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 848 2024-04-25T03:41:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 849 2024-04-25T03:41:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 851 2024-04-25T03:41:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 850 2024-04-25T03:41:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 853 2024-04-25T03:41:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 852 2024-04-25T03:41:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 854 2024-04-25T03:41:53.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 855 2024-04-25T03:41:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:41:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:41:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:41:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:41:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:41:54.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 856 2024-04-25T03:41:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 857 2024-04-25T03:41:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:41:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:41:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:41:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:41:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:41:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:41:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 858 2024-04-25T03:41:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:41:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:41:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:41:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:41:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:41:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 859 2024-04-25T03:41:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 861 2024-04-25T03:41:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 860 2024-04-25T03:41:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 863 2024-04-25T03:41:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 862 2024-04-25T03:41:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:41:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:41:58.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:41:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:41:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 864 2024-04-25T03:41:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 865 2024-04-25T03:41:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:41:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:41:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:41:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:41:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:41:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:41:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:41:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:41:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:41:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 866 2024-04-25T03:41:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 867 2024-04-25T03:41:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:41:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 868 2024-04-25T03:42:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 869 2024-04-25T03:42:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:00.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 870 2024-04-25T03:42:01.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 871 2024-04-25T03:42:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:01.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:02.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:02.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 872 2024-04-25T03:42:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 873 2024-04-25T03:42:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 874 2024-04-25T03:42:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 875 2024-04-25T03:42:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 877 2024-04-25T03:42:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 876 2024-04-25T03:42:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 879 2024-04-25T03:42:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 878 2024-04-25T03:42:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:05.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 880 2024-04-25T03:42:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 881 2024-04-25T03:42:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:06.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 883 2024-04-25T03:42:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 882 2024-04-25T03:42:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:07.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 884 2024-04-25T03:42:08.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 885 2024-04-25T03:42:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:08.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 886 2024-04-25T03:42:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 887 2024-04-25T03:42:09.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:09.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 888 2024-04-25T03:42:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 889 2024-04-25T03:42:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 890 2024-04-25T03:42:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 891 2024-04-25T03:42:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 892 2024-04-25T03:42:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 893 2024-04-25T03:42:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 895 2024-04-25T03:42:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 894 2024-04-25T03:42:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:13.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:13.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 897 2024-04-25T03:42:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 896 2024-04-25T03:42:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:14.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:14.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:14.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 898 2024-04-25T03:42:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 899 2024-04-25T03:42:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 901 2024-04-25T03:42:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 900 2024-04-25T03:42:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:16.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:16.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 903 2024-04-25T03:42:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 902 2024-04-25T03:42:17.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:17.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 904 2024-04-25T03:42:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 905 2024-04-25T03:42:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 906 2024-04-25T03:42:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 907 2024-04-25T03:42:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 908 2024-04-25T03:42:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 909 2024-04-25T03:42:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 910 2024-04-25T03:42:21.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 911 2024-04-25T03:42:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 912 2024-04-25T03:42:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 913 2024-04-25T03:42:23.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:42:23.023+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667a83400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:42:23.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:23.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:42:23.031+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657b3000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:42:23.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:23.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 914 2024-04-25T03:42:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 915 2024-04-25T03:42:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 916 2024-04-25T03:42:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 917 2024-04-25T03:42:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:24.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 918 2024-04-25T03:42:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 919 2024-04-25T03:42:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 921 2024-04-25T03:42:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 920 2024-04-25T03:42:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 922 2024-04-25T03:42:27.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 923 2024-04-25T03:42:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:27.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:27.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:27.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:27.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 924 2024-04-25T03:42:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 925 2024-04-25T03:42:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 926 2024-04-25T03:42:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 927 2024-04-25T03:42:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 929 2024-04-25T03:42:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 928 2024-04-25T03:42:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 931 2024-04-25T03:42:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 930 2024-04-25T03:42:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 934 2024-04-25T03:42:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 935 2024-04-25T03:42:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:32.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 933 2024-04-25T03:42:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 932 2024-04-25T03:42:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:33.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:34.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 936 2024-04-25T03:42:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 937 2024-04-25T03:42:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 939 2024-04-25T03:42:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 938 2024-04-25T03:42:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:36.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 942 2024-04-25T03:42:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 940 2024-04-25T03:42:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 943 2024-04-25T03:42:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 941 2024-04-25T03:42:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:36.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 944 2024-04-25T03:42:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 945 2024-04-25T03:42:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 947 2024-04-25T03:42:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 946 2024-04-25T03:42:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 952 2024-04-25T03:42:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 951 2024-04-25T03:42:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 949 2024-04-25T03:42:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 948 2024-04-25T03:42:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 953 2024-04-25T03:42:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 950 2024-04-25T03:42:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:42:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:42:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:42:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 954 2024-04-25T03:42:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 955 2024-04-25T03:42:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:42:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:42:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 956 2024-04-25T03:42:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 958 2024-04-25T03:42:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:42:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 959 2024-04-25T03:42:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 957 2024-04-25T03:42:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 960 2024-04-25T03:42:45.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 961 2024-04-25T03:42:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:45.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 962 2024-04-25T03:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:42:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:42:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:42:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 963 2024-04-25T03:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 965 2024-04-25T03:42:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 964 2024-04-25T03:42:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:42:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:42:47.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:42:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:42:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:42:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:42:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66570 2024-04-25T03:42:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:42:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:42:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:42:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 966 2024-04-25T03:42:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 967 2024-04-25T03:42:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:42:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66704 2024-04-25T03:42:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 969 2024-04-25T03:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 968 2024-04-25T03:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66838 2024-04-25T03:42:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66992 2024-04-25T03:42:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 970 2024-04-25T03:42:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 971 2024-04-25T03:42:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67147 2024-04-25T03:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:42:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:42:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67302 2024-04-25T03:42:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67457 2024-04-25T03:42:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 972 2024-04-25T03:42:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:42:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67632 2024-04-25T03:42:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:42:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67808 2024-04-25T03:42:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:42:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:42:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66437 2024-04-25T03:45:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66571 2024-04-25T03:45:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66705 2024-04-25T03:45:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66839 2024-04-25T03:45:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 66993 2024-04-25T03:45:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67148 2024-04-25T03:45:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67303 2024-04-25T03:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67458 2024-04-25T03:45:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67633 2024-04-25T03:45:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:45:31.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 67809 2024-04-25T03:45:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:45:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T03:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T03:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T03:45:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T03:45:33.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T03:45:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T03:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T03:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T03:45:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T03:45:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T03:45:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T03:45:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T03:45:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:45:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73801 2024-04-25T03:45:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:45:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:45:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:45:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:45:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:45:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:45:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:45:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:45:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:45:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:45:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:45:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=a6bc9202fdab81a76536cc616593c291 2024-04-25T03:46:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T03:46:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=d69b59f8f48bb1093d24d2ce2673be5f 2024-04-25T03:46:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T03:46:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=05ad0a051d2c5dc899078563080b3e65 2024-04-25T03:46:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T03:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T03:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T03:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T03:46:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T03:46:30.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T03:46:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T03:46:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=b6a3dc8f0ed78595bc0c194388a06a96 2024-04-25T03:46:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T03:46:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=d8c555e747bc1bd8a4a19ec42afaebd7 2024-04-25T03:46:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T03:46:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=4f99370459ea940022c14eb018c2b124 2024-04-25T03:46:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T03:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T03:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T03:46:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T03:46:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T03:46:30.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T03:46:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T03:46:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=4bf7a7be2f51068e43a48eecf407c27b 2024-04-25T03:46:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T03:46:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=f3991cb20798c92414821b4ab8bd07df 2024-04-25T03:46:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T03:46:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T03:46:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T03:46:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=f42b4f3bd92872e6fc6cfefebf0d47c9 2024-04-25T03:46:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T03:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T03:46:30.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:30.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T03:46:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T03:46:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T03:46:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=542bac5c60bf083fd6525d483954d5a8 2024-04-25T03:46:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T03:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T03:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T03:46:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:31.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T03:46:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T03:46:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:46:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T03:46:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T03:46:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T03:46:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T03:46:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:46:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d680229-5269-461c-bcc8-835d37bcb581 2024-04-25T03:46:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:32.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:46:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:46:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:46:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:33.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:46:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:46:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:46:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:46:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:46:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:46:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:46:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:46:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:33.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:46:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:46:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:46:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:46:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ae95fc20-1bd3-40e6-a713-6cef123e8a98 2024-04-25T03:46:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e77e2d29-1e62-423a-ac2a-2802074ee891 2024-04-25T03:46:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:46:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 634e8d7a-89d2-4c21-afd8-664d6872f4ef 2024-04-25T03:46:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:33.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:33.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:46:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9dc2fa16-a672-4b8a-bd96-3f12debbec37 2024-04-25T03:46:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce9a2183-b3fb-4968-905c-afc3395a175e 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:46:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4baef160-bd07-4521-ad9d-62c56fa53b00 2024-04-25T03:46:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:46:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17eed5eb-d5ab-413a-be73-f100e37f507c 2024-04-25T03:46:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:46:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:46:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d680229-5269-461c-bcc8-835d37bcb581 2024-04-25T03:46:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:46:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 72053786-3993-4b5c-8420-674034616008 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f4cd445f-0346-4445-abe3-a66a688d8165 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:46:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:46:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:46:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:46:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:46:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:46:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ae95fc20-1bd3-40e6-a713-6cef123e8a98 2024-04-25T03:46:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:46:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:46:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:46:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:46:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:35.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:46:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:46:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e77e2d29-1e62-423a-ac2a-2802074ee891 2024-04-25T03:46:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:46:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 634e8d7a-89d2-4c21-afd8-664d6872f4ef 2024-04-25T03:46:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9dc2fa16-a672-4b8a-bd96-3f12debbec37 2024-04-25T03:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:46:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce9a2183-b3fb-4968-905c-afc3395a175e 2024-04-25T03:46:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:46:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17eed5eb-d5ab-413a-be73-f100e37f507c 2024-04-25T03:46:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:46:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4baef160-bd07-4521-ad9d-62c56fa53b00 2024-04-25T03:46:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:35.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:46:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f4cd445f-0346-4445-abe3-a66a688d8165 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 72053786-3993-4b5c-8420-674034616008 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:46:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:46:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d680229-5269-461c-bcc8-835d37bcb581 2024-04-25T03:46:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:39.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:46:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:46:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:46:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:46:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:46:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:46:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ae95fc20-1bd3-40e6-a713-6cef123e8a98 2024-04-25T03:46:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:46:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:46:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:46:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:39.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:46:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:46:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:46:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e77e2d29-1e62-423a-ac2a-2802074ee891 2024-04-25T03:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:46:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 634e8d7a-89d2-4c21-afd8-664d6872f4ef 2024-04-25T03:46:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:46:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9dc2fa16-a672-4b8a-bd96-3f12debbec37 2024-04-25T03:46:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:46:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce9a2183-b3fb-4968-905c-afc3395a175e 2024-04-25T03:46:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:46:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17eed5eb-d5ab-413a-be73-f100e37f507c 2024-04-25T03:46:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:39.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:46:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4baef160-bd07-4521-ad9d-62c56fa53b00 2024-04-25T03:46:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:46:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:46:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:46:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f4cd445f-0346-4445-abe3-a66a688d8165 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:46:41.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 72053786-3993-4b5c-8420-674034616008 2024-04-25T03:46:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:46:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:46:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:16' 2024-04-25T03:46:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:41.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:46:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:46:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:46:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:46:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:46:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d680229-5269-461c-bcc8-835d37bcb581 2024-04-25T03:46:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:46.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:46.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:46.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:46.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:46:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:46:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T03:46:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:46:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:46:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:46:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ae95fc20-1bd3-40e6-a713-6cef123e8a98 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:46:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:46:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:46:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:46:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:46:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:46:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:46:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e77e2d29-1e62-423a-ac2a-2802074ee891 2024-04-25T03:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:46:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 634e8d7a-89d2-4c21-afd8-664d6872f4ef 2024-04-25T03:46:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:46:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9dc2fa16-a672-4b8a-bd96-3f12debbec37 2024-04-25T03:46:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce9a2183-b3fb-4968-905c-afc3395a175e 2024-04-25T03:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:47.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17eed5eb-d5ab-413a-be73-f100e37f507c 2024-04-25T03:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:47.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:46:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4baef160-bd07-4521-ad9d-62c56fa53b00 2024-04-25T03:46:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:46:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:46:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:46:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:46:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T03:46:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:46:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:46:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:46:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:46:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T03:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:46:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:46:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:46:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d680229-5269-461c-bcc8-835d37bcb581 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T03:46:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:46:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:46:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:46:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:46:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:48.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T03:46:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:46:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T03:46:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T03:46:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T03:46:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T03:46:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:46:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ae95fc20-1bd3-40e6-a713-6cef123e8a98 2024-04-25T03:46:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T03:46:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T03:46:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T03:46:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T03:46:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:46:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T03:46:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T03:46:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T03:46:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T03:46:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T03:46:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T03:46:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T03:46:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:46:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e77e2d29-1e62-423a-ac2a-2802074ee891 2024-04-25T03:46:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:46:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:46:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T03:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T03:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 634e8d7a-89d2-4c21-afd8-664d6872f4ef 2024-04-25T03:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:46:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9dc2fa16-a672-4b8a-bd96-3f12debbec37 2024-04-25T03:46:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4baef160-bd07-4521-ad9d-62c56fa53b00 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce9a2183-b3fb-4968-905c-afc3395a175e 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:46:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17eed5eb-d5ab-413a-be73-f100e37f507c 2024-04-25T03:46:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:46:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:46:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T03:46:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:46:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:46:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:46:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:46:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:46:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:46:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:46:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:46:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:46:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T03:46:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T03:46:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:46:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T03:46:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:46:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:46:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:46:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:46:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T03:46:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f4cd445f-0346-4445-abe3-a66a688d8165 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T03:46:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:46:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T03:46:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 72053786-3993-4b5c-8420-674034616008 2024-04-25T03:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:49.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:49.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:49.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:46:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:46:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:46:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:46:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T03:46:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:46:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T03:46:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T03:46:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T03:46:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T03:46:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T03:46:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T03:46:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T03:46:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T03:46:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:46:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T03:46:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T03:46:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T03:46:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:46:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:46:50.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T03:46:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:46:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:46:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T03:46:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:46:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T03:46:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:46:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T03:46:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T03:46:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T03:46:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:46:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:46:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T03:46:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:46:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f4cd445f-0346-4445-abe3-a66a688d8165 2024-04-25T03:46:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:46:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 72053786-3993-4b5c-8420-674034616008 2024-04-25T03:46:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:46:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:46:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:46:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:46:46' 2024-04-25T03:46:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:46:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:46:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:46:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:46:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:46:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:46:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:46:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:46:51.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:46:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:46:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 03:46:46' 2024-04-25T03:46:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:46:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:46:46' 2024-04-25T03:46:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:46:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:46:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:46:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:46:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:46:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary472053786-3993-4b5c-8420-674034616008up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:46:46' 2024-04-25T03:46:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:46:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:46:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:46:46' 2024-04-25T03:46:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:46:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T03:46:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:46:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T03:46:51.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:46:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:46:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T03:46:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:46:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:46:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:46:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:46:51.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T03:46:53.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:46:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:53.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T03:46:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:46:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:46:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T03:46:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T03:46:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:46:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:46:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T03:46:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:46:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:47:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=4bf7a7be2f51068e43a48eecf407c27b 2024-04-25T03:47:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T03:47:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=05ad0a051d2c5dc899078563080b3e65 2024-04-25T03:47:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T03:47:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 4bf7a7be2f51068e43a48eecf407c27b != \4\b\f\7\a\7\b\e\2\f\5\1\0\6\8\e\4\3\a\4\8\e\e\c\f\4\0\7\c\2\7\b ]] 2024-04-25T03:47:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 05ad0a051d2c5dc899078563080b3e65 != \0\5\a\d\0\a\0\5\1\d\2\c\5\d\c\8\9\9\0\7\8\5\6\3\0\8\0\b\3\e\6\5 ]] 2024-04-25T03:47:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=f3991cb20798c92414821b4ab8bd07df 2024-04-25T03:47:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T03:47:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ f3991cb20798c92414821b4ab8bd07df != \f\3\9\9\1\c\b\2\0\7\9\8\c\9\2\4\1\4\8\2\1\b\4\a\b\8\b\d\0\7\d\f ]] 2024-04-25T03:47:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=4f99370459ea940022c14eb018c2b124 2024-04-25T03:47:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T03:47:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=542bac5c60bf083fd6525d483954d5a8 2024-04-25T03:47:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T03:47:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 4f99370459ea940022c14eb018c2b124 != \4\f\9\9\3\7\0\4\5\9\e\a\9\4\0\0\2\2\c\1\4\e\b\0\1\8\c\2\b\1\2\4 ]] 2024-04-25T03:47:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 542bac5c60bf083fd6525d483954d5a8 != \5\4\2\b\a\c\5\c\6\0\b\f\0\8\3\f\d\6\5\2\5\d\4\8\3\9\5\4\d\5\a\8 ]] 2024-04-25T03:47:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=f42b4f3bd92872e6fc6cfefebf0d47c9 2024-04-25T03:47:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T03:47:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ f42b4f3bd92872e6fc6cfefebf0d47c9 != \f\4\2\b\4\f\3\b\d\9\2\8\7\2\e\6\f\c\6\c\f\e\f\e\b\f\0\d\4\7\c\9 ]] 2024-04-25T03:47:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=b6a3dc8f0ed78595bc0c194388a06a96 2024-04-25T03:47:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T03:47:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=d8c555e747bc1bd8a4a19ec42afaebd7 2024-04-25T03:47:48.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T03:47:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ b6a3dc8f0ed78595bc0c194388a06a96 != \b\6\a\3\d\c\8\f\0\e\d\7\8\5\9\5\b\c\0\c\1\9\4\3\8\8\a\0\6\a\9\6 ]] 2024-04-25T03:47:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ d8c555e747bc1bd8a4a19ec42afaebd7 != \d\8\c\5\5\5\e\7\4\7\b\c\1\b\d\8\a\4\a\1\9\e\c\4\2\a\f\a\e\b\d\7 ]] 2024-04-25T03:47:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73802 2024-04-25T03:47:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73804 2024-04-25T03:47:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73806 2024-04-25T03:47:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=d69b59f8f48bb1093d24d2ce2673be5f 2024-04-25T03:47:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T03:47:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ d69b59f8f48bb1093d24d2ce2673be5f != \d\6\9\b\5\9\f\8\f\4\8\b\b\1\0\9\3\d\2\4\d\2\c\e\2\6\7\3\b\e\5\f ]] 2024-04-25T03:47:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73808 2024-04-25T03:47:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73810 2024-04-25T03:47:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73812 2024-04-25T03:47:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73814 2024-04-25T03:47:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73816 2024-04-25T03:47:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=a6bc9202fdab81a76536cc616593c291 2024-04-25T03:47:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ a6bc9202fdab81a76536cc616593c291 != \a\6\b\c\9\2\0\2\f\d\a\b\8\1\a\7\6\5\3\6\c\c\6\1\6\5\9\3\c\2\9\1 ]] 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 73817 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T03:47:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:47:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:47:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T03:47:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:47:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:47:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:47:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:47:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:47:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:47:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce9a2183-b3fb-4968-905c-afc3395a175e 2024-04-25T03:47:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:47:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3158835.2,"bytes_per_snapshot":63176704.0,"last_snapshot_bytes":63176704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:47:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:47:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:47:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:47:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:47:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:47:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3158835.2,"bytes_per_snapshot":63176704.0,"last_snapshot_bytes":63176704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:47:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:47:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:47:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:47:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:47:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:47:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:47:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:47:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:47:51.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:47:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:47:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:47:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce9a2183-b3fb-4968-905c-afc3395a175eup+replayingreplaying, {"bytes_per_second":3158835.2,"bytes_per_snapshot":63176704.0,"last_snapshot_bytes":63176704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+replayingreplaying, {"bytes_per_second":3615334.4,"bytes_per_snapshot":72306688.0,"last_snapshot_bytes":72306688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+replayingreplaying, {"bytes_per_second":3205734.4,"bytes_per_snapshot":64114688.0,"last_snapshot_bytes":64114688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary472053786-3993-4b5c-8420-674034616008up+replayingreplaying, {"bytes_per_second":3238297.6,"bytes_per_snapshot":64765952.0,"last_snapshot_bytes":64765952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+replayingreplaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+replayingreplaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+replayingreplaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:47:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:47:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:47:51.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:47:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:47:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:47:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:47:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:47:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:47:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T03:47:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:47:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:47:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T03:47:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T03:47:51.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:47:51.449+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966447fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:47:51.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:47:51.449+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/ce9a2183-b3fb-4968-905c-afc3395a175e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:47:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:47:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T03:47:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:47:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:47:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:47:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T03:47:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T03:47:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:47:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:47:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T03:47:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:47:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:47:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:47:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T03:47:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:47:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:47:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:47:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:47:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:47:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:47:54.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:47:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e77e2d29-1e62-423a-ac2a-2802074ee891 2024-04-25T03:47:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:47:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3615334.4,"bytes_per_snapshot":72306688.0,"last_snapshot_bytes":72306688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:47:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:47:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:47:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:47:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:47:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:47:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3615334.4,"bytes_per_snapshot":72306688.0,"last_snapshot_bytes":72306688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:47:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:47:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:47:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:47:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:47:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:47:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:47:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:47:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:47:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:47:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:47:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:47:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:47:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:47:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:47:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:47:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary2e77e2d29-1e62-423a-ac2a-2802074ee891up+replayingreplaying, {"bytes_per_second":3615334.4,"bytes_per_snapshot":72306688.0,"last_snapshot_bytes":72306688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+replayingreplaying, {"bytes_per_second":3205734.4,"bytes_per_snapshot":64114688.0,"last_snapshot_bytes":64114688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary472053786-3993-4b5c-8420-674034616008up+replayingreplaying, {"bytes_per_second":3238297.6,"bytes_per_snapshot":64765952.0,"last_snapshot_bytes":64765952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+replayingreplaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+replayingreplaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+replayingreplaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:47:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:47:54.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:47:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:47:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:47:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:47:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:47:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:47:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:47:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T03:47:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:47:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:47:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T03:47:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T03:47:54.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:47:54.724+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c56ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:47:54.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:47:54.724+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdac80 [3/e77e2d29-1e62-423a-ac2a-2802074ee891] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:47:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:47:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T03:47:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:47:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:47:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:47:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T03:47:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T03:47:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:47:56.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:47:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:47:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:47:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:47:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:47:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:47:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9dc2fa16-a672-4b8a-bd96-3f12debbec37 2024-04-25T03:47:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:47:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3205734.4,"bytes_per_snapshot":64114688.0,"last_snapshot_bytes":64114688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:47:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:47:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:47:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:47:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:47:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:47:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3205734.4,"bytes_per_snapshot":64114688.0,"last_snapshot_bytes":64114688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:47:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:47:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:47:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:47:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:47:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:47:57.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:47:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:47:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:47:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:47:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:47:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:47:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:47:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:47:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:47:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:47:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary39dc2fa16-a672-4b8a-bd96-3f12debbec37up+replayingreplaying, {"bytes_per_second":3205734.4,"bytes_per_snapshot":64114688.0,"last_snapshot_bytes":64114688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary472053786-3993-4b5c-8420-674034616008up+replayingreplaying, {"bytes_per_second":3238297.6,"bytes_per_snapshot":64765952.0,"last_snapshot_bytes":64765952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+replayingreplaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+replayingreplaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+replayingreplaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:47:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:47:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:47:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:47:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:47:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:47:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:47:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:47:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:47:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T03:47:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:47:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:47:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T03:47:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T03:47:57.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:47:57.712+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596644801a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:47:57.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:47:57.712+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdaa00 [3/9dc2fa16-a672-4b8a-bd96-3f12debbec37] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:47:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:47:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T03:47:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:47:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:47:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:47:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T03:47:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:47:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:47:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:47:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:48:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:48:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 72053786-3993-4b5c-8420-674034616008 2024-04-25T03:48:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:48:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3238297.6,"bytes_per_snapshot":64765952.0,"last_snapshot_bytes":64765952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"} 2024-04-25T03:48:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:48:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:48:00.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:48:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:48:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:48:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3238297.6,"bytes_per_snapshot":64765952.0,"last_snapshot_bytes":64765952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"} 2024-04-25T03:48:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:48:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:00.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:48:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:48:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:48:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:48:00.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:48:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:48:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary472053786-3993-4b5c-8420-674034616008up+replayingreplaying, {"bytes_per_second":3238297.6,"bytes_per_snapshot":64765952.0,"last_snapshot_bytes":64765952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+replayingreplaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+replayingreplaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+replayingreplaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:48:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:48:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:48:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:48:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:48:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:48:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T03:48:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:48:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:00.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T03:48:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T03:48:00.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:00.714+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c631e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:48:00.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:00.714+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/72053786-3993-4b5c-8420-674034616008] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:48:02.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:48:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T03:48:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:48:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T03:48:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T03:48:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:48:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T03:48:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:48:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:48:03.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ae95fc20-1bd3-40e6-a713-6cef123e8a98 2024-04-25T03:48:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:48:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:48:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:48:03.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:48:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:48:03.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:48:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:48:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:48:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:48:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:48:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:48:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:48:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:48:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:48:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:48:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary5ae95fc20-1bd3-40e6-a713-6cef123e8a98up+replayingreplaying, {"bytes_per_second":3614515.2,"bytes_per_snapshot":72290304.0,"last_snapshot_bytes":72290304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+replayingreplaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+replayingreplaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:48:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:48:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:48:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:48:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:48:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:48:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T03:48:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:48:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T03:48:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T03:48:03.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:03.593+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664d10b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:48:03.597 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:03.593+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb680 [3/ae95fc20-1bd3-40e6-a713-6cef123e8a98] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:48:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:48:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T03:48:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:48:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T03:48:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:05.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:05.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:48:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T03:48:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:48:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:48:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d680229-5269-461c-bcc8-835d37bcb581 2024-04-25T03:48:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:48:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"} 2024-04-25T03:48:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:48:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:48:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:48:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:48:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:48:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"} 2024-04-25T03:48:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:48:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:48:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:48:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:48:06.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:48:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:48:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:06.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:48:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary61d680229-5269-461c-bcc8-835d37bcb581up+replayingreplaying, {"bytes_per_second":3632947.2,"bytes_per_snapshot":72658944.0,"last_snapshot_bytes":72658944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016808,"remote_snapshot_timestamp":1714016808,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+replayingreplaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:48:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:48:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:48:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:48:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:48:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:48:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T03:48:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:48:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T03:48:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T03:48:06.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:06.878+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966450a000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:48:06.881 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:06.878+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/1d680229-5269-461c-bcc8-835d37bcb581] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:48:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:48:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T03:48:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:48:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T03:48:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:08.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:48:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T03:48:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:48:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:48:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 634e8d7a-89d2-4c21-afd8-664d6872f4ef 2024-04-25T03:48:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:48:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:48:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:48:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:48:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:48:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:48:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:48:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:48:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:48:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:48:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:48:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:48:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary7634e8d7a-89d2-4c21-afd8-664d6872f4efup+replayingreplaying, {"bytes_per_second":3610828.8,"bytes_per_snapshot":72216576.0,"last_snapshot_bytes":72216576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:48:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:48:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:48:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:48:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:48:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:48:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T03:48:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:48:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T03:48:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T03:48:09.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:09.574+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966447e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:48:09.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:09.574+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508000 [3/634e8d7a-89d2-4c21-afd8-664d6872f4ef] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:48:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:48:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T03:48:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:48:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T03:48:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T03:48:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:48:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T03:48:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:48:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:48:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 17eed5eb-d5ab-413a-be73-f100e37f507c 2024-04-25T03:48:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:48:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:48:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:48:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:48:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:48:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:48:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:48:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:48:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:48:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:48:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:48:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:48:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:48:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:48:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:48:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary817eed5eb-d5ab-413a-be73-f100e37f507cup+replayingreplaying, {"bytes_per_second":3128115.2,"bytes_per_snapshot":62562304.0,"last_snapshot_bytes":62562304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:48:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:48:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:48:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:48:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:48:12.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:48:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:12.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T03:48:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:48:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T03:48:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T03:48:12.586 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:12.583+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668499d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:48:12.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:12.583+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/17eed5eb-d5ab-413a-be73-f100e37f507c] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:48:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:48:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T03:48:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:48:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T03:48:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T03:48:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:15.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:48:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:48:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f4cd445f-0346-4445-abe3-a66a688d8165 2024-04-25T03:48:15.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:48:15.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"} 2024-04-25T03:48:15.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:48:15.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:47:46' 2024-04-25T03:48:15.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:48:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:48:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:48:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"} 2024-04-25T03:48:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:48:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:15.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:48:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:48:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:48:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:48:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 03:47:46' 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:47:46' 2024-04-25T03:48:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:15.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:48:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":3589939.2,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46image-primary9f4cd445f-0346-4445-abe3-a66a688d8165up+replayingreplaying, {"bytes_per_second":3376947.2,"bytes_per_snapshot":67538944.0,"last_snapshot_bytes":67538944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016812,"remote_snapshot_timestamp":1714016812,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:47:46' 2024-04-25T03:48:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:48:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:48:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:47:46' 2024-04-25T03:48:15.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:48:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:48:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:48:15.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T03:48:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:48:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T03:48:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T03:48:15.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:15.743+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c5d860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:48:15.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:15.743+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/f4cd445f-0346-4445-abe3-a66a688d8165] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:48:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:48:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T03:48:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:48:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T03:48:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T03:48:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:48:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T03:48:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:48:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T03:48:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:48:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:48:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4baef160-bd07-4521-ad9d-62c56fa53b00 2024-04-25T03:48:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:48:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:48:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:48:16' 2024-04-25T03:48:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:48:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:48:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:48:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"} 2024-04-25T03:48:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:48:18.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:48:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:48:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:48:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:48:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:48:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:48:16' 2024-04-25T03:48:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 03:48:16' 2024-04-25T03:48:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:18.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:48:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:48:16' 2024-04-25T03:48:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:48:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:48:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:48:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary104baef160-bd07-4521-ad9d-62c56fa53b00up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71798784.0,"last_snapshot_bytes":71798784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714016809,"remote_snapshot_timestamp":1714016809,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:48:16' 2024-04-25T03:48:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:48:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:48:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:48:16' 2024-04-25T03:48:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:48:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:48:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:48:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:48:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:48:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T03:48:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:48:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T03:48:18.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T03:48:18.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:18.581+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664563040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:48:18.585 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:18.581+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508780 [3/4baef160-bd07-4521-ad9d-62c56fa53b00] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:48:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:48:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T03:48:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:48:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:48:20.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T03:48:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T03:48:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T03:48:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:20.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T03:48:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T03:48:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1f6b7eb29f5a 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1f6b7eb29f5a 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:20 2024 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:20 2024 2024-04-25T03:48:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:20 2024 2024-04-25T03:48:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:48:21.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T03:48:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T03:48:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T03:48:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 27efc378-31d5-43e8-a24b-3997b65808d4 2024-04-25T03:48:21.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T03:48:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T03:48:21.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T03:48:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:21.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:48:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T03:48:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T03:48:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T03:48:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T03:48:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T03:48:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:21.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T03:48:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T03:48:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:22.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 994 2024-04-25T03:48:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:22.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T03:48:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T03:48:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1f7bea4b3696 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1f7bea4b3696 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:21 2024 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:21 2024 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:21 2024 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:48:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T03:48:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T03:48:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T03:48:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 2ec519d7-5995-40e3-91c4-469908cbee99 2024-04-25T03:48:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T03:48:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T03:48:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T03:48:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T03:48:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:22.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T03:48:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T03:48:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T03:48:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 996 2024-04-25T03:48:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T03:48:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T03:48:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1f89c59f649f 2024-04-25T03:48:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1f89c59f649f 2024-04-25T03:48:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:22 2024 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:22 2024 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:22 2024 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:48:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T03:48:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T03:48:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T03:48:23.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: d9428e78-2c6f-45cf-9597-d82f2e9c07dc 2024-04-25T03:48:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T03:48:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T03:48:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T03:48:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:48:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T03:48:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T03:48:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T03:48:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T03:48:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 998 2024-04-25T03:48:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T03:48:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T03:48:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1f987a3e631b 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1f987a3e631b 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:23 2024 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:23 2024 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:23 2024 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:48:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T03:48:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T03:48:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:24.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T03:48:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 65bc203e-a25a-4b5d-b0d5-effea2f8fb71 2024-04-25T03:48:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T03:48:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T03:48:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T03:48:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:24.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T03:48:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T03:48:25.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1000 2024-04-25T03:48:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T03:48:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T03:48:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1fa654748032 2024-04-25T03:48:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1fa654748032 2024-04-25T03:48:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:24 2024 2024-04-25T03:48:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:24 2024 2024-04-25T03:48:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:24 2024 2024-04-25T03:48:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: a86d1657-ebcb-4c3e-a174-b22c84fbb2a7 2024-04-25T03:48:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T03:48:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T03:48:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T03:48:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 4429935d-872b-4864-9cd7-048ece2a525b 2024-04-25T03:48:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:25.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T03:48:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T03:48:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T03:48:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T03:48:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T03:48:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T03:48:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T03:48:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:26.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1003 2024-04-25T03:48:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1002 2024-04-25T03:48:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1fb8e701a437 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1fb8e701a437 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:25 2024 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:25 2024 2024-04-25T03:48:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:25 2024 2024-04-25T03:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:48:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:26.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T03:48:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T03:48:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T03:48:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ed9315cc-6a80-4643-b500-436e189c5aa3 2024-04-25T03:48:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:26.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T03:48:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T03:48:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T03:48:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T03:48:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T03:48:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:48:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T03:48:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T03:48:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T03:48:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T03:48:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T03:48:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:26.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T03:48:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T03:48:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1005 2024-04-25T03:48:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1006 2024-04-25T03:48:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1fca16f2021 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1fca16f2021 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:27.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:26 2024 2024-04-25T03:48:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:26 2024 2024-04-25T03:48:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:26 2024 2024-04-25T03:48:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:48:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T03:48:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T03:48:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T03:48:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 4332a7bf-2dc3-4970-a3f3-9246dc7eaab3 2024-04-25T03:48:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:27.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T03:48:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T03:48:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:27.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:27.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:27.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T03:48:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T03:48:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T03:48:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T03:48:28.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1009 2024-04-25T03:48:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1008 2024-04-25T03:48:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T03:48:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1fdc2be9a942 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1fdc2be9a942 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:27 2024 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:27 2024 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:27 2024 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:48:28.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T03:48:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T03:48:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T03:48:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 44e5ae61-0107-4475-92d9-9e56000e0cbc 2024-04-25T03:48:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T03:48:28.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T03:48:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T03:48:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T03:48:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T03:48:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T03:48:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T03:48:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T03:48:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T03:48:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1011 2024-04-25T03:48:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1013 2024-04-25T03:48:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:29.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:29.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T03:48:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 1feeee6a8c9 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.1feeee6a8c9 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:28 2024 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:28 2024 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:28 2024 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:48:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T03:48:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T03:48:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T03:48:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 58de92c9-d292-4d99-976c-cc6b6b136f0d 2024-04-25T03:48:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:29.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T03:48:29.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T03:48:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T03:48:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T03:48:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T03:48:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T03:48:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:48:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T03:48:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:48:29.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T03:48:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T03:48:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:48:30.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1015 2024-04-25T03:48:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1014 2024-04-25T03:48:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1016 2024-04-25T03:48:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T03:48:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2003cf155d71 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2003cf155d71 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:48:29 2024 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:48:29 2024 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:48:29 2024 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:48:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:48:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:48:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:48:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:48:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:48:30.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T03:48:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T03:48:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:48:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T03:48:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ca2d9a32-deca-4120-ae57-c446cb81422b 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:48:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:48:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:48:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:48:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:48:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T03:48:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T03:48:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:48:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T03:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T03:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78143 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T03:48:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T03:48:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:48:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T03:48:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1020 2024-04-25T03:48:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1019 2024-04-25T03:48:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1018 2024-04-25T03:48:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:32.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1022 2024-04-25T03:48:32.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1021 2024-04-25T03:48:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:33.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1024 2024-04-25T03:48:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1023 2024-04-25T03:48:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1027 2024-04-25T03:48:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:48:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:48:34.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1026 2024-04-25T03:48:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1025 2024-04-25T03:48:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1028 2024-04-25T03:48:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1030 2024-04-25T03:48:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:35.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1029 2024-04-25T03:48:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1032 2024-04-25T03:48:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1031 2024-04-25T03:48:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:37.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1034 2024-04-25T03:48:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1033 2024-04-25T03:48:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1037 2024-04-25T03:48:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1036 2024-04-25T03:48:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1035 2024-04-25T03:48:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:48:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:38.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:48:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1038 2024-04-25T03:48:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1039 2024-04-25T03:48:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1040 2024-04-25T03:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:40.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1042 2024-04-25T03:48:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1041 2024-04-25T03:48:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1043 2024-04-25T03:48:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:41.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1044 2024-04-25T03:48:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1045 2024-04-25T03:48:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1047 2024-04-25T03:48:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:48:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:48:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1046 2024-04-25T03:48:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1049 2024-04-25T03:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1048 2024-04-25T03:48:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1051 2024-04-25T03:48:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1050 2024-04-25T03:48:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1052 2024-04-25T03:48:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1053 2024-04-25T03:48:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1054 2024-04-25T03:48:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1055 2024-04-25T03:48:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:48:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:48:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:47.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1059 2024-04-25T03:48:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1056 2024-04-25T03:48:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1058 2024-04-25T03:48:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1057 2024-04-25T03:48:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1060 2024-04-25T03:48:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1061 2024-04-25T03:48:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:48:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:48:50.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:50.209+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596677d3c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:48:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1063 2024-04-25T03:48:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1062 2024-04-25T03:48:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:51.183 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:51.179+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665a42400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:48:51.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:51.185+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667530c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:48:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1065 2024-04-25T03:48:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1064 2024-04-25T03:48:52.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1067 2024-04-25T03:48:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1066 2024-04-25T03:48:53.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:48:53.102+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596656d4400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:48:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1068 2024-04-25T03:48:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1069 2024-04-25T03:48:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1070 2024-04-25T03:48:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1071 2024-04-25T03:48:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1072 2024-04-25T03:48:55.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1073 2024-04-25T03:48:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:55.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:48:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:48:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:48:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:48:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:48:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1074 2024-04-25T03:48:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1075 2024-04-25T03:48:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:48:57.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:48:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:48:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:48:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:48:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:48:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:48:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:48:57.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:48:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1077 2024-04-25T03:48:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:48:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:48:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:48:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1076 2024-04-25T03:48:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1079 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:48:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:48:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1078 2024-04-25T03:48:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:48:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:48:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:48:59.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:48:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:48:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:48:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:48:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:48:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:48:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:48:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1083 2024-04-25T03:48:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1081 2024-04-25T03:49:00.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1080 2024-04-25T03:49:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:00.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1084 2024-04-25T03:49:00.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1085 2024-04-25T03:49:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:01.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1082 2024-04-25T03:49:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1087 2024-04-25T03:49:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1088 2024-04-25T03:49:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:02.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:03.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1086 2024-04-25T03:49:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1090 2024-04-25T03:49:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:03.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1089 2024-04-25T03:49:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1093 2024-04-25T03:49:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1092 2024-04-25T03:49:04.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1094 2024-04-25T03:49:04.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1091 2024-04-25T03:49:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:04.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:04.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1095 2024-04-25T03:49:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:05.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:06.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1097 2024-04-25T03:49:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:07.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1099 2024-04-25T03:49:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1098 2024-04-25T03:49:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1096 2024-04-25T03:49:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:08.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1103 2024-04-25T03:49:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1100 2024-04-25T03:49:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1102 2024-04-25T03:49:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1104 2024-04-25T03:49:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1101 2024-04-25T03:49:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1105 2024-04-25T03:49:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1106 2024-04-25T03:49:10.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:10.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1107 2024-04-25T03:49:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:12.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1109 2024-04-25T03:49:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:12.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1108 2024-04-25T03:49:13.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1111 2024-04-25T03:49:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1112 2024-04-25T03:49:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:13.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:13.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1110 2024-04-25T03:49:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1114 2024-04-25T03:49:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1113 2024-04-25T03:49:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:14.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1116 2024-04-25T03:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1115 2024-04-25T03:49:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1117 2024-04-25T03:49:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:17.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1119 2024-04-25T03:49:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:18.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1120 2024-04-25T03:49:18.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1121 2024-04-25T03:49:18.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1118 2024-04-25T03:49:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1122 2024-04-25T03:49:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1123 2024-04-25T03:49:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1124 2024-04-25T03:49:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1126 2024-04-25T03:49:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:20.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1125 2024-04-25T03:49:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:21.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:21.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1131 2024-04-25T03:49:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1127 2024-04-25T03:49:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1128 2024-04-25T03:49:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:23.135 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:49:23.132+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665734c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:49:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:24.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1132 2024-04-25T03:49:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1133 2024-04-25T03:49:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1134 2024-04-25T03:49:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1135 2024-04-25T03:49:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1129 2024-04-25T03:49:25.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1130 2024-04-25T03:49:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:26.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:27.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1139 2024-04-25T03:49:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:27.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:27.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1137 2024-04-25T03:49:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1140 2024-04-25T03:49:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1136 2024-04-25T03:49:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1141 2024-04-25T03:49:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1142 2024-04-25T03:49:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:30.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:30.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1144 2024-04-25T03:49:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1138 2024-04-25T03:49:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1143 2024-04-25T03:49:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1145 2024-04-25T03:49:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1148 2024-04-25T03:49:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1147 2024-04-25T03:49:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1146 2024-04-25T03:49:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1149 2024-04-25T03:49:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1150 2024-04-25T03:49:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1151 2024-04-25T03:49:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1153 2024-04-25T03:49:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1152 2024-04-25T03:49:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1156 2024-04-25T03:49:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1154 2024-04-25T03:49:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1155 2024-04-25T03:49:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1158 2024-04-25T03:49:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1157 2024-04-25T03:49:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1159 2024-04-25T03:49:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:39.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1160 2024-04-25T03:49:39.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1161 2024-04-25T03:49:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1163 2024-04-25T03:49:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1164 2024-04-25T03:49:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1162 2024-04-25T03:49:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:41.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1166 2024-04-25T03:49:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1165 2024-04-25T03:49:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:41.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1167 2024-04-25T03:49:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1168 2024-04-25T03:49:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1169 2024-04-25T03:49:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1171 2024-04-25T03:49:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:44.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1170 2024-04-25T03:49:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1173 2024-04-25T03:49:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1172 2024-04-25T03:49:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1174 2024-04-25T03:49:45.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1176 2024-04-25T03:49:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1175 2024-04-25T03:49:46.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1178 2024-04-25T03:49:47.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1177 2024-04-25T03:49:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:48.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:49:48.121+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596655fd800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:49:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:48.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1179 2024-04-25T03:49:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1181 2024-04-25T03:49:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1180 2024-04-25T03:49:50.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1183 2024-04-25T03:49:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1184 2024-04-25T03:49:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1182 2024-04-25T03:49:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1185 2024-04-25T03:49:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1186 2024-04-25T03:49:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1188 2024-04-25T03:49:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1187 2024-04-25T03:49:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1189 2024-04-25T03:49:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:49:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:49:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:49:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1190 2024-04-25T03:49:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:49:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1191 2024-04-25T03:49:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1193 2024-04-25T03:49:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1192 2024-04-25T03:49:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:49:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:49:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:49:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:49:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:49:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:49:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:49:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:49:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:49:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1195 2024-04-25T03:49:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1196 2024-04-25T03:49:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:49:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:57.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:49:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:49:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1199 2024-04-25T03:49:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1198 2024-04-25T03:49:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:57.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:49:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:49:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:49:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1194 2024-04-25T03:49:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1197 2024-04-25T03:49:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:49:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:49:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1201 2024-04-25T03:49:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1200 2024-04-25T03:49:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:49:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:49:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:49:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:49:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:49:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:49:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:49:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:49:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:49:59.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1203 2024-04-25T03:50:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1202 2024-04-25T03:50:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:00.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1205 2024-04-25T03:50:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1204 2024-04-25T03:50:01.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:01.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1206 2024-04-25T03:50:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1209 2024-04-25T03:50:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:02.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:02.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1207 2024-04-25T03:50:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:03.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:50:03.124+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667778c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:50:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1208 2024-04-25T03:50:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1211 2024-04-25T03:50:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1210 2024-04-25T03:50:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1216 2024-04-25T03:50:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1213 2024-04-25T03:50:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1212 2024-04-25T03:50:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1214 2024-04-25T03:50:07.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1215 2024-04-25T03:50:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1218 2024-04-25T03:50:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1217 2024-04-25T03:50:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1219 2024-04-25T03:50:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:08.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1220 2024-04-25T03:50:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1221 2024-04-25T03:50:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1222 2024-04-25T03:50:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:10.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:10.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1224 2024-04-25T03:50:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1223 2024-04-25T03:50:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:11.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1225 2024-04-25T03:50:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1226 2024-04-25T03:50:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1228 2024-04-25T03:50:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1229 2024-04-25T03:50:12.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1227 2024-04-25T03:50:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1231 2024-04-25T03:50:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1230 2024-04-25T03:50:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:15.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1232 2024-04-25T03:50:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1233 2024-04-25T03:50:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1234 2024-04-25T03:50:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1236 2024-04-25T03:50:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1235 2024-04-25T03:50:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1237 2024-04-25T03:50:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1238 2024-04-25T03:50:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1239 2024-04-25T03:50:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:18.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1241 2024-04-25T03:50:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1240 2024-04-25T03:50:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:19.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:19.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:20.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1243 2024-04-25T03:50:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1244 2024-04-25T03:50:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1242 2024-04-25T03:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1247 2024-04-25T03:50:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1245 2024-04-25T03:50:21.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1246 2024-04-25T03:50:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:22.163 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:50:22.160+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667b1dc00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:50:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:22.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1249 2024-04-25T03:50:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1248 2024-04-25T03:50:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1251 2024-04-25T03:50:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1250 2024-04-25T03:50:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1252 2024-04-25T03:50:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1253 2024-04-25T03:50:25.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1256 2024-04-25T03:50:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1255 2024-04-25T03:50:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1259 2024-04-25T03:50:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1258 2024-04-25T03:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1254 2024-04-25T03:50:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1257 2024-04-25T03:50:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1260 2024-04-25T03:50:29.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1261 2024-04-25T03:50:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:29.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1263 2024-04-25T03:50:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1262 2024-04-25T03:50:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1264 2024-04-25T03:50:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1265 2024-04-25T03:50:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1266 2024-04-25T03:50:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1267 2024-04-25T03:50:33.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1269 2024-04-25T03:50:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:33.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1268 2024-04-25T03:50:34.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1270 2024-04-25T03:50:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1271 2024-04-25T03:50:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1276 2024-04-25T03:50:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1272 2024-04-25T03:50:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:36.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1273 2024-04-25T03:50:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1275 2024-04-25T03:50:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1274 2024-04-25T03:50:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1277 2024-04-25T03:50:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1279 2024-04-25T03:50:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:38.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:39.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1278 2024-04-25T03:50:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1280 2024-04-25T03:50:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1281 2024-04-25T03:50:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1282 2024-04-25T03:50:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1283 2024-04-25T03:50:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1284 2024-04-25T03:50:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:40.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:40.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:40.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1285 2024-04-25T03:50:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1286 2024-04-25T03:50:41.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:50:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:50:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:50:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1287 2024-04-25T03:50:42.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:50:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:50:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:50:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:50:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:50:42.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:50:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1288 2024-04-25T03:50:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:50:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:50:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:50:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1289 2024-04-25T03:50:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1292 2024-04-25T03:50:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:50:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:50:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1290 2024-04-25T03:50:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:50:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:44.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:50:44.171+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559668ca0000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:50:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:45.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1291 2024-04-25T03:50:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:45.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:45.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:50:45.181+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667a75c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:50:45.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:50:45.184+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665a43400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:50:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1294 2024-04-25T03:50:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1293 2024-04-25T03:50:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:50:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:50:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:50:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78277 2024-04-25T03:50:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1296 2024-04-25T03:50:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1295 2024-04-25T03:50:47.153 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:50:47.150+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665a5b800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:50:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78411 2024-04-25T03:50:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1297 2024-04-25T03:50:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1298 2024-04-25T03:50:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:50:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:50:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:50:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1299 2024-04-25T03:50:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:50:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:50:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:50:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78545 2024-04-25T03:50:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1300 2024-04-25T03:50:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78699 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78854 2024-04-25T03:50:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:50:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:50:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:50:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:50:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:50:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:50:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79009 2024-04-25T03:50:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1301 2024-04-25T03:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1302 2024-04-25T03:50:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79164 2024-04-25T03:50:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:50:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:53.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79339 2024-04-25T03:50:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:50:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79515 2024-04-25T03:50:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:50:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:50:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78144 2024-04-25T03:53:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78278 2024-04-25T03:53:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78412 2024-04-25T03:53:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78546 2024-04-25T03:53:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78700 2024-04-25T03:53:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 78855 2024-04-25T03:53:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79010 2024-04-25T03:53:27.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:27.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79165 2024-04-25T03:53:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79340 2024-04-25T03:53:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:53:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 79516 2024-04-25T03:53:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T03:53:30.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T03:53:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:53:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T03:53:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T03:53:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T03:53:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T03:53:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T03:53:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T03:53:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T03:53:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T03:53:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T03:53:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T03:53:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T03:53:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T03:53:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T03:53:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T03:53:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T03:53:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T03:53:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T03:53:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T03:53:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T03:53:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:53:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85510 2024-04-25T03:53:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T03:53:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T03:53:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T03:53:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T03:53:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T03:53:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T03:53:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:53:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:53:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:53:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:53:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:53:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:53:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:53:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:53:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:53:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:53:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:53:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=600fc27c89199ca66e21f39c53fe3406 2024-04-25T03:54:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T03:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T03:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T03:54:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=53b793f281de0b8f77e80208a244122c 2024-04-25T03:54:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T03:54:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=21234380a1c523cbf8719a9ee68ab96a 2024-04-25T03:54:26.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T03:54:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T03:54:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T03:54:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T03:54:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T03:54:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=6332b4e2b2956ed6540bfe126c0dcf94 2024-04-25T03:54:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T03:54:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T03:54:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T03:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=f9c03f108af35e1c855b04001631c858 2024-04-25T03:54:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T03:54:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T03:54:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:28.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T03:54:28.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=9a4b92e92204acdc682063029fa9a19a 2024-04-25T03:54:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T03:54:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T03:54:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T03:54:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:28.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T03:54:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=a977ceecb4c13b3db4cab1ae3f93c2db 2024-04-25T03:54:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T03:54:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T03:54:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T03:54:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=30e51497c7738f4d19a103a2e708a3ab 2024-04-25T03:54:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T03:54:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=197a8fd1f71d88eea123661a208d9ff3 2024-04-25T03:54:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T03:54:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=3372401aba6b97cc621e5836d63f8df5 2024-04-25T03:54:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T03:54:29.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:54:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:29.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T03:54:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T03:54:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T03:54:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T03:54:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:29.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T03:54:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:54:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:54:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:30.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T03:54:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:30.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:54:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:54:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:54:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:54:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:54:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:54:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:30.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:54:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:54:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:30.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:30.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:54:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:54:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:31.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:31.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:31.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:31.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:31.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:54:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:54:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:54:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T03:54:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T03:54:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:54:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:54:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:32.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:54:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:54:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:54:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:54:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:54:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:54:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:54:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:54:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:54:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:54:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:54:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:54:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:54:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:54:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:54:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:32.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:54:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:54:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:54:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:54:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:54:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a86d1657-ebcb-4c3e-a174-b22c84fbb2a7 2024-04-25T03:54:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T03:54:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:54:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:54:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:54:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:33.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:54:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:54:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:34.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:54:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:34.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:54:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:54:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:54:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:54:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a86d1657-ebcb-4c3e-a174-b22c84fbb2a7 2024-04-25T03:54:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:35.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T03:54:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:54:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:54:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:54:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:54:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:35.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:54:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:54:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:36.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:54:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:54:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:54:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:54:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:54:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:54:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:54:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:54:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:36.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:36.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:54:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:54:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:36.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:54:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:54:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:54:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:54:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:54:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:54:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:54:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a86d1657-ebcb-4c3e-a174-b22c84fbb2a7 2024-04-25T03:54:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:54:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:54:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:54:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:54:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:54:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:54:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:54:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:54:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:54:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:54:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:54:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:54:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:44.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:54:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:54:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:54:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:54:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:54:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:54:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:54:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:45.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:54:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:54:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:54:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:54:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:54:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:54:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:54:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:46.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:54:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:16' 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T03:54:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T03:54:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T03:54:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a86d1657-ebcb-4c3e-a174-b22c84fbb2a7 2024-04-25T03:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:54:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:54:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T03:54:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T03:54:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:54:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a86d1657-ebcb-4c3e-a174-b22c84fbb2a7 2024-04-25T03:54:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:48.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:48.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:54:48.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:54:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T03:54:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:54:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T03:54:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T03:54:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T03:54:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T03:54:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:54:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:54:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:54:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:54:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:51.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:54:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T03:54:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T03:54:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:54:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:54:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:52.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:54:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:54:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:54:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:54:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T03:54:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:54:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:54:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:54:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T03:54:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:54:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:54:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:54:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:54:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:54:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T03:54:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:54:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:54:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T03:54:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:54:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T03:54:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T03:54:53.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T03:54:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T03:54:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:54:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:54:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:54:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:53.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:54:53.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:54:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T03:54:53.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T03:54:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T03:54:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T03:54:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:54:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:54:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T03:54:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T03:54:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:54:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:54:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:54:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:54:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:54:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:54:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:53.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:54:54.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:54:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T03:54:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:54:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T03:54:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:54:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:54:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T03:54:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:54:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:54:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T03:54:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:54:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T03:54:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T03:54:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T03:54:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:54:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T03:54:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:54:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:54:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:54:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:54:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:54.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:54.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T03:54:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T03:54:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:54:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:54:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:54:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:54:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:54:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:54:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T03:54:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:54.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T03:54:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:54:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:54:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:54:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:54.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:54.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T03:54:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T03:54:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T03:54:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T03:54:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T03:54:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:55.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T03:54:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T03:54:55.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T03:54:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T03:54:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T03:54:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T03:54:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T03:54:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T03:54:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T03:54:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T03:54:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T03:54:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T03:54:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T03:54:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T03:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:55.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:54:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:54:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:54:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:54:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T03:54:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T03:54:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:54:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:54:46' 2024-04-25T03:54:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T03:54:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T03:54:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:54:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T03:54:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:54:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:54:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:54:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:54:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:54:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:54:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 03:54:46' 2024-04-25T03:54:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T03:54:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:54:46' 2024-04-25T03:54:56.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:54:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:54:56.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T03:54:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:54:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:54:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T03:54:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:54:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T03:54:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:54:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:54:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T03:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:54:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T03:54:56.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 03:54:46' 2024-04-25T03:54:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:54:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:54:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:54:46' 2024-04-25T03:54:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:54:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T03:54:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T03:54:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T03:54:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T03:54:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:54:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T03:54:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:54:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:54:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:54:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T03:54:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T03:54:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T03:54:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T03:54:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T03:54:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:54:57.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:54:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T03:54:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T03:54:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T03:54:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T03:54:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:54:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T03:54:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T03:54:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T03:54:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T03:55:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=21234380a1c523cbf8719a9ee68ab96a 2024-04-25T03:55:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T03:55:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 21234380a1c523cbf8719a9ee68ab96a != \2\1\2\3\4\3\8\0\a\1\c\5\2\3\c\b\f\8\7\1\9\a\9\e\e\6\8\a\b\9\6\a ]] 2024-04-25T03:55:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=600fc27c89199ca66e21f39c53fe3406 2024-04-25T03:55:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T03:55:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 600fc27c89199ca66e21f39c53fe3406 != \6\0\0\f\c\2\7\c\8\9\1\9\9\c\a\6\6\e\2\1\f\3\9\c\5\3\f\e\3\4\0\6 ]] 2024-04-25T03:55:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=6332b4e2b2956ed6540bfe126c0dcf94 2024-04-25T03:55:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T03:55:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 6332b4e2b2956ed6540bfe126c0dcf94 != \6\3\3\2\b\4\e\2\b\2\9\5\6\e\d\6\5\4\0\b\f\e\1\2\6\c\0\d\c\f\9\4 ]] 2024-04-25T03:55:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=f9c03f108af35e1c855b04001631c858 2024-04-25T03:55:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T03:55:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=9a4b92e92204acdc682063029fa9a19a 2024-04-25T03:55:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T03:55:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=a977ceecb4c13b3db4cab1ae3f93c2db 2024-04-25T03:55:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T03:55:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ f9c03f108af35e1c855b04001631c858 != \f\9\c\0\3\f\1\0\8\a\f\3\5\e\1\c\8\5\5\b\0\4\0\0\1\6\3\1\c\8\5\8 ]] 2024-04-25T03:55:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 9a4b92e92204acdc682063029fa9a19a != \9\a\4\b\9\2\e\9\2\2\0\4\a\c\d\c\6\8\2\0\6\3\0\2\9\f\a\9\a\1\9\a ]] 2024-04-25T03:55:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ a977ceecb4c13b3db4cab1ae3f93c2db != \a\9\7\7\c\e\e\c\b\4\c\1\3\b\3\d\b\4\c\a\b\1\a\e\3\f\9\3\c\2\d\b ]] 2024-04-25T03:55:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=53b793f281de0b8f77e80208a244122c 2024-04-25T03:55:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T03:55:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 53b793f281de0b8f77e80208a244122c != \5\3\b\7\9\3\f\2\8\1\d\e\0\b\8\f\7\7\e\8\0\2\0\8\a\2\4\4\1\2\2\c ]] 2024-04-25T03:55:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=197a8fd1f71d88eea123661a208d9ff3 2024-04-25T03:55:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T03:55:52.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 197a8fd1f71d88eea123661a208d9ff3 != \1\9\7\a\8\f\d\1\f\7\1\d\8\8\e\e\a\1\2\3\6\6\1\a\2\0\8\d\9\f\f\3 ]] 2024-04-25T03:55:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=30e51497c7738f4d19a103a2e708a3ab 2024-04-25T03:55:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T03:55:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 30e51497c7738f4d19a103a2e708a3ab != \3\0\e\5\1\4\9\7\c\7\7\3\8\f\4\d\1\9\a\1\0\3\a\2\e\7\0\8\a\3\a\b ]] 2024-04-25T03:55:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=3372401aba6b97cc621e5836d63f8df5 2024-04-25T03:55:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T03:55:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 3372401aba6b97cc621e5836d63f8df5 != \3\3\7\2\4\0\1\a\b\a\6\b\9\7\c\c\6\2\1\e\5\8\3\6\d\6\3\f\8\d\f\5 ]] 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85511 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85513 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85515 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85517 2024-04-25T03:55:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85519 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85521 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85523 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85525 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 85527 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:55:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T03:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:55:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:55:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:55:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T03:55:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T03:55:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a14d9bbb-4b78-4417-baac-7421cdaa4f41 2024-04-25T03:55:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:55:54.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3656908.8,"bytes_per_snapshot":73138176.0,"last_snapshot_bytes":73138176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"} 2024-04-25T03:55:54.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:55:54.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:55:54.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:55:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:55:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:55:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3656908.8,"bytes_per_snapshot":73138176.0,"last_snapshot_bytes":73138176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"} 2024-04-25T03:55:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:55:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:55:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:55:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:55:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:55:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:55:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:55:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:55:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:55:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:55:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:55:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:55:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:55:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:55:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:55:55.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:55:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1a14d9bbb-4b78-4417-baac-7421cdaa4f41up+replayingreplaying, {"bytes_per_second":3656908.8,"bytes_per_snapshot":73138176.0,"last_snapshot_bytes":73138176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+replayingreplaying, {"bytes_per_second":3175014.4,"bytes_per_snapshot":63500288.0,"last_snapshot_bytes":63500288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+replayingreplaying, {"bytes_per_second":3184435.2,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+replayingreplaying, {"bytes_per_second":3678822.4,"bytes_per_snapshot":73576448.0,"last_snapshot_bytes":73576448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+replayingreplaying, {"bytes_per_second":3689062.4,"bytes_per_snapshot":73781248.0,"last_snapshot_bytes":73781248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017289,"remote_snapshot_timestamp":1714017289,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+replayingreplaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+replayingreplaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:55:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:55:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T03:55:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:55:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T03:55:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:55:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:55:55.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:55:55.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:55:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:55:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:55:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T03:55:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:55:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:55:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T03:55:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T03:55:55.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:55:55.368+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c5a1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:55:55.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:55:55.368+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/a14d9bbb-4b78-4417-baac-7421cdaa4f41] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:55:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:55:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T03:55:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:55:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:55:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:55:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T03:55:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:55:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T03:55:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:55:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:55:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:55:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:55:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:55:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T03:55:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T03:55:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b35fa092-154a-4d0d-be02-081266f1d3f6 2024-04-25T03:55:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:55:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3175014.4,"bytes_per_snapshot":63500288.0,"last_snapshot_bytes":63500288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:55:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:55:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:55:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:55:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:55:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:55:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3175014.4,"bytes_per_snapshot":63500288.0,"last_snapshot_bytes":63500288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:55:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:55:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:55:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:55:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:55:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:55:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:55:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:55:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:55:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary2b35fa092-154a-4d0d-be02-081266f1d3f6up+replayingreplaying, {"bytes_per_second":3175014.4,"bytes_per_snapshot":63500288.0,"last_snapshot_bytes":63500288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+replayingreplaying, {"bytes_per_second":3184435.2,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+replayingreplaying, {"bytes_per_second":3678822.4,"bytes_per_snapshot":73576448.0,"last_snapshot_bytes":73576448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+replayingreplaying, {"bytes_per_second":3689062.4,"bytes_per_snapshot":73781248.0,"last_snapshot_bytes":73781248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017289,"remote_snapshot_timestamp":1714017289,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+replayingreplaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+replayingreplaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:55:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:55:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T03:55:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:55:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T03:55:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:55:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:55:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:55:58.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T03:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:55:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T03:55:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T03:55:58.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:55:58.873+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664f04820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:55:58.877 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:55:58.873+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508780 [3/b35fa092-154a-4d0d-be02-081266f1d3f6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T03:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T03:56:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T03:56:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T03:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T03:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: aef8ec94-fd7d-4d1f-91cb-90dbee641ec3 2024-04-25T03:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3184435.2,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:56:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3184435.2,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:56:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary3aef8ec94-fd7d-4d1f-91cb-90dbee641ec3up+replayingreplaying, {"bytes_per_second":3184435.2,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+replayingreplaying, {"bytes_per_second":3678822.4,"bytes_per_snapshot":73576448.0,"last_snapshot_bytes":73576448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+replayingreplaying, {"bytes_per_second":3689062.4,"bytes_per_snapshot":73781248.0,"last_snapshot_bytes":73781248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017289,"remote_snapshot_timestamp":1714017289,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+replayingreplaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+replayingreplaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:56:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T03:56:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T03:56:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T03:56:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T03:56:01.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T03:56:01.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:01.875+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667bc7ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:01.880 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:01.875+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/aef8ec94-fd7d-4d1f-91cb-90dbee641ec3] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T03:56:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T03:56:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T03:56:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T03:56:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T03:56:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5079fb81-5870-4652-9a02-9c93d21d90f7 2024-04-25T03:56:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3678822.4,"bytes_per_snapshot":73576448.0,"last_snapshot_bytes":73576448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:56:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3678822.4,"bytes_per_snapshot":73576448.0,"last_snapshot_bytes":73576448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:56:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:56:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary45079fb81-5870-4652-9a02-9c93d21d90f7up+replayingreplaying, {"bytes_per_second":3678822.4,"bytes_per_snapshot":73576448.0,"last_snapshot_bytes":73576448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+replayingreplaying, {"bytes_per_second":3689062.4,"bytes_per_snapshot":73781248.0,"last_snapshot_bytes":73781248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017289,"remote_snapshot_timestamp":1714017289,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+replayingreplaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+replayingreplaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:56:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T03:56:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T03:56:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T03:56:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T03:56:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T03:56:04.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:04.662+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668cd24e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:04.667 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:04.662+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508000 [3/5079fb81-5870-4652-9a02-9c93d21d90f7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T03:56:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:06.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T03:56:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:06.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T03:56:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T03:56:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T03:56:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a86d1657-ebcb-4c3e-a174-b22c84fbb2a7 2024-04-25T03:56:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3689062.4,"bytes_per_snapshot":73781248.0,"last_snapshot_bytes":73781248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017289,"remote_snapshot_timestamp":1714017289,"replay_state":"idle"} 2024-04-25T03:56:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:07.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:56:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3689062.4,"bytes_per_snapshot":73781248.0,"last_snapshot_bytes":73781248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017289,"remote_snapshot_timestamp":1714017289,"replay_state":"idle"} 2024-04-25T03:56:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:56:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:56:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary5a86d1657-ebcb-4c3e-a174-b22c84fbb2a7up+replayingreplaying, {"bytes_per_second":3689062.4,"bytes_per_snapshot":73781248.0,"last_snapshot_bytes":73781248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017289,"remote_snapshot_timestamp":1714017289,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+replayingreplaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+replayingreplaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T03:56:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T03:56:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T03:56:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T03:56:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T03:56:07.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:07.694+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966447f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:07.699 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:07.694+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/a86d1657-ebcb-4c3e-a174-b22c84fbb2a7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T03:56:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T03:56:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:09.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T03:56:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T03:56:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T03:56:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5a652cd-9edc-4f53-827b-c89494d309ba 2024-04-25T03:56:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"} 2024-04-25T03:56:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:56:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"} 2024-04-25T03:56:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:10.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:10.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:10.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:56:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary6c5a652cd-9edc-4f53-827b-c89494d309baup+replayingreplaying, {"bytes_per_second":3679436.8,"bytes_per_snapshot":73588736.0,"last_snapshot_bytes":73588736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017293,"remote_snapshot_timestamp":1714017293,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+replayingreplaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:56:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T03:56:10.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T03:56:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T03:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T03:56:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T03:56:10.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:10.849+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667bc6b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:10.854 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:10.849+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/c5a652cd-9edc-4f53-827b-c89494d309ba] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:12.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T03:56:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T03:56:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T03:56:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:56:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T03:56:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T03:56:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0 2024-04-25T03:56:13.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"} 2024-04-25T03:56:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:56:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"} 2024-04-25T03:56:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:13.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:56:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:56:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary73d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0up+replayingreplaying, {"bytes_per_second":3600793.6,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:56:13.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T03:56:13.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:13.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T03:56:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:13.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T03:56:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T03:56:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T03:56:13.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:13.673+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668498b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:13.678 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:13.673+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/3d5d5fff-0def-4b1c-89d6-8a5fe8ea3fb0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T03:56:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:15.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T03:56:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:16.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T03:56:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T03:56:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 60ef0426-e040-4f33-a38f-2952fc136cd0 2024-04-25T03:56:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"} 2024-04-25T03:56:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:55:46' 2024-04-25T03:56:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"} 2024-04-25T03:56:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:16.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 03:55:46' 2024-04-25T03:56:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:55:46' 2024-04-25T03:56:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":3292160.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary860ef0426-e040-4f33-a38f-2952fc136cd0up+replayingreplaying, {"bytes_per_second":3266355.2,"bytes_per_snapshot":65327104.0,"last_snapshot_bytes":65327104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017297,"remote_snapshot_timestamp":1714017297,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":3314278.4,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:55:46' 2024-04-25T03:56:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T03:56:16.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:55:46' 2024-04-25T03:56:16.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T03:56:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:16.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T03:56:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:16.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T03:56:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T03:56:16.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:16.711+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667878b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:16.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:16.711+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/60ef0426-e040-4f33-a38f-2952fc136cd0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:16.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:16.803+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966447d6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:16.808 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:16.803+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/60ef0426-e040-4f33-a38f-2952fc136cd0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T03:56:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T03:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T03:56:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:56:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T03:56:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T03:56:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T03:56:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 04da554a-9098-4a21-973e-5af628c1b929 2024-04-25T03:56:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:56:16' 2024-04-25T03:56:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:56:16' 2024-04-25T03:56:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 03:56:16' 2024-04-25T03:56:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:56:16' 2024-04-25T03:56:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:56:16image-primary904da554a-9098-4a21-973e-5af628c1b929up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66285568.0,"last_snapshot_bytes":66285568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:56:16' 2024-04-25T03:56:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:19.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T03:56:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:56:16' 2024-04-25T03:56:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T03:56:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:19.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T03:56:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T03:56:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T03:56:19.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:19.878+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667bc6340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:19.882 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:19.878+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/04da554a-9098-4a21-973e-5af628c1b929] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:21.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T03:56:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:21.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T03:56:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T03:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T03:56:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T03:56:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T03:56:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2aff9ed3-1034-4b89-ad5a-01e7ce1c5042 2024-04-25T03:56:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T03:56:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T03:56:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 03:56:16' 2024-04-25T03:56:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T03:56:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T03:56:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T03:56:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"} 2024-04-25T03:56:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T03:56:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T03:56:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T03:56:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T03:56:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T03:56:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T03:56:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:56:16' 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 03:56:16' 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 03:56:16' 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T03:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T03:56:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary102aff9ed3-1034-4b89-ad5a-01e7ce1c5042up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65843200.0,"last_snapshot_bytes":65843200,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017295,"remote_snapshot_timestamp":1714017295,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 03:56:16' 2024-04-25T03:56:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T03:56:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T03:56:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 03:56:16' 2024-04-25T03:56:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T03:56:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T03:56:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T03:56:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T03:56:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T03:56:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T03:56:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T03:56:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T03:56:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T03:56:22.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:22.712+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966447dd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T03:56:22.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:22.712+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508a00 [3/2aff9ed3-1034-4b89-ad5a-01e7ce1c5042] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T03:56:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T03:56:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T03:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T03:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T03:56:24.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T03:56:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T03:56:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T03:56:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T03:56:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T03:56:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T03:56:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T03:56:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T03:56:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T03:56:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2468897b9fb0 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2468897b9fb0 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:24 2024 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:24 2024 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:24 2024 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 1c188272-a2b9-4499-badd-37a62f0b1308 2024-04-25T03:56:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:25.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T03:56:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T03:56:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T03:56:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 1e8c6360-4723-413a-a568-2bebc5cc267b 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T03:56:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T03:56:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T03:56:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T03:56:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T03:56:26.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T03:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1324 2024-04-25T03:56:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T03:56:26.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2478c247f446 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2478c247f446 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:25 2024 2024-04-25T03:56:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:25 2024 2024-04-25T03:56:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:25 2024 2024-04-25T03:56:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 54b1f6ce-6058-4f9a-923a-4ecd1a629c84 2024-04-25T03:56:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T03:56:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T03:56:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T03:56:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: a9d8cb51-d817-4de2-9c37-7361d4070b7a 2024-04-25T03:56:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T03:56:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T03:56:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T03:56:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T03:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:27.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T03:56:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:27.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1326 2024-04-25T03:56:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T03:56:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2486def53383 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2486def53383 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:26 2024 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:26 2024 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:26 2024 2024-04-25T03:56:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: c5bb0fb4-fda6-43ff-9412-65e1c45bbb03 2024-04-25T03:56:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:27.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T03:56:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T03:56:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T03:56:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 69fa5ca1-f6fc-42f7-b355-768b1b653ae6 2024-04-25T03:56:27.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:27.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T03:56:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T03:56:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T03:56:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T03:56:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T03:56:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T03:56:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T03:56:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1328 2024-04-25T03:56:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2493b2a146bb 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2493b2a146bb 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:27 2024 2024-04-25T03:56:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:27 2024 2024-04-25T03:56:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:27 2024 2024-04-25T03:56:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca 2024-04-25T03:56:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T03:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T03:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T03:56:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: df1d0eb0-58cc-448a-a58c-52bf16f480d2 2024-04-25T03:56:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:28.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T03:56:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T03:56:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T03:56:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T03:56:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T03:56:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T03:56:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T03:56:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T03:56:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1330 2024-04-25T03:56:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T03:56:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T03:56:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 24a2d7965c87 2024-04-25T03:56:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.24a2d7965c87 2024-04-25T03:56:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:29.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:28 2024 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:28 2024 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:28 2024 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: eab25d1c-b433-4102-92de-d533d0647beb 2024-04-25T03:56:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T03:56:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T03:56:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T03:56:29.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: f119589b-5f4e-4b4a-9711-19e3c503f7ca 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:29.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T03:56:29.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T03:56:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T03:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T03:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T03:56:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T03:56:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:29.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T03:56:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T03:56:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T03:56:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1333 2024-04-25T03:56:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1332 2024-04-25T03:56:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T03:56:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 24b4a88fd112 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.24b4a88fd112 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:30.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:29 2024 2024-04-25T03:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:29 2024 2024-04-25T03:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:29 2024 2024-04-25T03:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: a9cb7225-1db7-4afc-8106-36f27b6ee1c3 2024-04-25T03:56:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T03:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T03:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T03:56:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: a420170f-248a-4bee-8912-e41fd92c8542 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T03:56:30.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T03:56:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T03:56:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:56:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T03:56:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T03:56:30.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T03:56:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T03:56:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1335 2024-04-25T03:56:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1336 2024-04-25T03:56:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T03:56:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 24c62987e89d 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.24c62987e89d 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:30 2024 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:30 2024 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:30 2024 2024-04-25T03:56:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 252b1098-64c5-4869-8ed4-6442d5f44803 2024-04-25T03:56:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T03:56:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T03:56:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T03:56:31.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: fe0789e4-30b2-4fd0-bffb-1984a4530a0a 2024-04-25T03:56:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T03:56:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T03:56:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:56:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T03:56:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T03:56:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:32.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T03:56:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:32.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:32.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1338 2024-04-25T03:56:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1339 2024-04-25T03:56:32.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:32.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T03:56:32.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 24d8aae24d09 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.24d8aae24d09 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:31 2024 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:31 2024 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:31 2024 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ffc8aa0e-7041-4b77-aba4-1b0cb879d619 2024-04-25T03:56:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T03:56:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T03:56:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T03:56:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 28964f42-4156-42de-a823-76336b3471af 2024-04-25T03:56:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:32.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T03:56:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T03:56:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:32.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T03:56:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T03:56:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T03:56:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T03:56:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T03:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T03:56:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T03:56:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1341 2024-04-25T03:56:33.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1342 2024-04-25T03:56:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T03:56:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:33.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 24ead5c4bf15 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.24ead5c4bf15 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:32 2024 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:32 2024 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:32 2024 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: f3c2a3ad-06eb-4864-b13b-dcd75558a220 2024-04-25T03:56:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:56:33.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T03:56:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T03:56:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T03:56:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 8e1008ca-d725-48c6-a15f-b4f03771a55e 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T03:56:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T03:56:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T03:56:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T03:56:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T03:56:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T03:56:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T03:56:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T03:56:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:33.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T03:56:33.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T03:56:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T03:56:34.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T03:56:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1345 2024-04-25T03:56:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1344 2024-04-25T03:56:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1346 2024-04-25T03:56:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T03:56:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:56:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:34.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 24ff2aadb080 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.24ff2aadb080 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 03:56:33 2024 2024-04-25T03:56:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 03:56:33 2024 2024-04-25T03:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 03:56:33 2024 2024-04-25T03:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T03:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T03:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: fd15e173-2ffd-404f-a02b-7b1a3d547f76 2024-04-25T03:56:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T03:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T03:56:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T03:56:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T03:56:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T03:56:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T03:56:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T03:56:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: b0e1a9a5-8b81-407b-8b56-c402c06fcb3f 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T03:56:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T03:56:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T03:56:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T03:56:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T03:56:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T03:56:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T03:56:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T03:56:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T03:56:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:56:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:56:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T03:56:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90010 2024-04-25T03:56:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T03:56:34.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T03:56:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T03:56:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T03:56:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1348 2024-04-25T03:56:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1350 2024-04-25T03:56:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1349 2024-04-25T03:56:35.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:56:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:56:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:56:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:56:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1352 2024-04-25T03:56:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1351 2024-04-25T03:56:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:56:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:56:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:56:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1353 2024-04-25T03:56:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1354 2024-04-25T03:56:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1356 2024-04-25T03:56:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:56:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1357 2024-04-25T03:56:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1355 2024-04-25T03:56:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:56:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:56:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1360 2024-04-25T03:56:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1358 2024-04-25T03:56:39.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1359 2024-04-25T03:56:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:56:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:56:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1361 2024-04-25T03:56:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1362 2024-04-25T03:56:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:56:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:56:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1364 2024-04-25T03:56:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1363 2024-04-25T03:56:41.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:56:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:56:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:56:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:56:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:56:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1366 2024-04-25T03:56:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1367 2024-04-25T03:56:42.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1365 2024-04-25T03:56:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1370 2024-04-25T03:56:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1369 2024-04-25T03:56:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1368 2024-04-25T03:56:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:56:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:56:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:43.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1371 2024-04-25T03:56:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1372 2024-04-25T03:56:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:56:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:56:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1374 2024-04-25T03:56:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1373 2024-04-25T03:56:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:56:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:56:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:56:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1375 2024-04-25T03:56:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:56:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:56:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:56:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:56:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1377 2024-04-25T03:56:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:46.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:56:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:56:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1378 2024-04-25T03:56:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1379 2024-04-25T03:56:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1376 2024-04-25T03:56:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:56:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:56:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1381 2024-04-25T03:56:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1380 2024-04-25T03:56:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1383 2024-04-25T03:56:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:56:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:56:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:56:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1382 2024-04-25T03:56:50.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1385 2024-04-25T03:56:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:56:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:56:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:56:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1384 2024-04-25T03:56:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1387 2024-04-25T03:56:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:56:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:56:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1386 2024-04-25T03:56:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:52.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1389 2024-04-25T03:56:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:56:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:56:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1388 2024-04-25T03:56:53.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1391 2024-04-25T03:56:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:56:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:56:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1390 2024-04-25T03:56:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:54.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:56:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:56:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:56:54.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T03:56:54.295+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657b3400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T03:56:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1393 2024-04-25T03:56:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1392 2024-04-25T03:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:56:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:56:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:56:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1395 2024-04-25T03:56:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:56:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:56:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:56:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1394 2024-04-25T03:56:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:56:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:56:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:56:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1397 2024-04-25T03:56:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1396 2024-04-25T03:56:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:56:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:56:57.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:56:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1399 2024-04-25T03:56:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:56:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:56:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:56:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1398 2024-04-25T03:56:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:56:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:56:58.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:56:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1401 2024-04-25T03:56:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:56:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:56:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:56:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1400 2024-04-25T03:56:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:56:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:56:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:56:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1403 2024-04-25T03:56:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:56:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:56:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:56:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:56:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:56:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:56:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1402 2024-04-25T03:57:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1405 2024-04-25T03:57:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1404 2024-04-25T03:57:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:01.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:01.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1407 2024-04-25T03:57:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:01.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:01.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:02.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1406 2024-04-25T03:57:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1409 2024-04-25T03:57:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1408 2024-04-25T03:57:03.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:03.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:03.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1411 2024-04-25T03:57:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1412 2024-04-25T03:57:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1410 2024-04-25T03:57:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1413 2024-04-25T03:57:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:05.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:05.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:05.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:05.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:05.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1415 2024-04-25T03:57:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1414 2024-04-25T03:57:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1417 2024-04-25T03:57:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:06.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1416 2024-04-25T03:57:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1421 2024-04-25T03:57:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1419 2024-04-25T03:57:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:07.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1418 2024-04-25T03:57:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1422 2024-04-25T03:57:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:08.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1420 2024-04-25T03:57:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1423 2024-04-25T03:57:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1425 2024-04-25T03:57:10.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1424 2024-04-25T03:57:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1428 2024-04-25T03:57:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:11.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1427 2024-04-25T03:57:11.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1426 2024-04-25T03:57:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1430 2024-04-25T03:57:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1429 2024-04-25T03:57:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1432 2024-04-25T03:57:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:14.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1431 2024-04-25T03:57:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1433 2024-04-25T03:57:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1435 2024-04-25T03:57:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:16.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1436 2024-04-25T03:57:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1434 2024-04-25T03:57:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1438 2024-04-25T03:57:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1437 2024-04-25T03:57:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1440 2024-04-25T03:57:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1439 2024-04-25T03:57:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1442 2024-04-25T03:57:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1441 2024-04-25T03:57:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1443 2024-04-25T03:57:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1446 2024-04-25T03:57:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1444 2024-04-25T03:57:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1448 2024-04-25T03:57:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1447 2024-04-25T03:57:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1450 2024-04-25T03:57:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:23.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1445 2024-04-25T03:57:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1449 2024-04-25T03:57:23.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1451 2024-04-25T03:57:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1452 2024-04-25T03:57:23.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1453 2024-04-25T03:57:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1455 2024-04-25T03:57:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:26.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1454 2024-04-25T03:57:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1457 2024-04-25T03:57:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:27.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1456 2024-04-25T03:57:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1460 2024-04-25T03:57:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1458 2024-04-25T03:57:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1462 2024-04-25T03:57:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1461 2024-04-25T03:57:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1459 2024-04-25T03:57:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1463 2024-04-25T03:57:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:30.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1465 2024-04-25T03:57:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:31.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1464 2024-04-25T03:57:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1467 2024-04-25T03:57:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1466 2024-04-25T03:57:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1469 2024-04-25T03:57:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:33.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1468 2024-04-25T03:57:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1470 2024-04-25T03:57:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1471 2024-04-25T03:57:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1473 2024-04-25T03:57:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1472 2024-04-25T03:57:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1475 2024-04-25T03:57:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:36.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1474 2024-04-25T03:57:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1477 2024-04-25T03:57:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1479 2024-04-25T03:57:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1476 2024-04-25T03:57:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1480 2024-04-25T03:57:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1478 2024-04-25T03:57:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1481 2024-04-25T03:57:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:38.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:40.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1482 2024-04-25T03:57:40.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1483 2024-04-25T03:57:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1485 2024-04-25T03:57:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1484 2024-04-25T03:57:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:41.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1488 2024-04-25T03:57:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1487 2024-04-25T03:57:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1486 2024-04-25T03:57:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:42.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1489 2024-04-25T03:57:42.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1490 2024-04-25T03:57:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1491 2024-04-25T03:57:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:43.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:44.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1492 2024-04-25T03:57:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1493 2024-04-25T03:57:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1495 2024-04-25T03:57:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1494 2024-04-25T03:57:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1497 2024-04-25T03:57:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1498 2024-04-25T03:57:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1496 2024-04-25T03:57:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1499 2024-04-25T03:57:48.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1501 2024-04-25T03:57:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1500 2024-04-25T03:57:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1503 2024-04-25T03:57:50.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1502 2024-04-25T03:57:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1505 2024-04-25T03:57:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1504 2024-04-25T03:57:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1508 2024-04-25T03:57:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1506 2024-04-25T03:57:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1507 2024-04-25T03:57:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1509 2024-04-25T03:57:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1510 2024-04-25T03:57:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1511 2024-04-25T03:57:52.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:53.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1513 2024-04-25T03:57:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1512 2024-04-25T03:57:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:57:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:57:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:57:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1515 2024-04-25T03:57:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:57:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:57:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:57:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:57:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:57:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:57:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:57:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:57:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:57:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1514 2024-04-25T03:57:56.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1518 2024-04-25T03:57:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1517 2024-04-25T03:57:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1516 2024-04-25T03:57:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:57:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:57:57.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:57:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:57:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:57:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:57:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1521 2024-04-25T03:57:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1520 2024-04-25T03:57:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1519 2024-04-25T03:57:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:57:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:57:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:57:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:57:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:57:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:57:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:57:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:57:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:57:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:57:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:57:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:57:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:57:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:57:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:57:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1522 2024-04-25T03:58:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1523 2024-04-25T03:58:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1525 2024-04-25T03:58:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1524 2024-04-25T03:58:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1528 2024-04-25T03:58:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1527 2024-04-25T03:58:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1526 2024-04-25T03:58:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1531 2024-04-25T03:58:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1529 2024-04-25T03:58:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1530 2024-04-25T03:58:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1532 2024-04-25T03:58:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1533 2024-04-25T03:58:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:05.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1535 2024-04-25T03:58:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1534 2024-04-25T03:58:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1537 2024-04-25T03:58:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1538 2024-04-25T03:58:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1536 2024-04-25T03:58:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1539 2024-04-25T03:58:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1540 2024-04-25T03:58:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1541 2024-04-25T03:58:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:08.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:09.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1542 2024-04-25T03:58:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1543 2024-04-25T03:58:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1545 2024-04-25T03:58:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1544 2024-04-25T03:58:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1548 2024-04-25T03:58:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1546 2024-04-25T03:58:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1547 2024-04-25T03:58:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:12.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1550 2024-04-25T03:58:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1551 2024-04-25T03:58:12.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1549 2024-04-25T03:58:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1552 2024-04-25T03:58:14.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1553 2024-04-25T03:58:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:15.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1555 2024-04-25T03:58:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1554 2024-04-25T03:58:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1557 2024-04-25T03:58:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1558 2024-04-25T03:58:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1556 2024-04-25T03:58:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1559 2024-04-25T03:58:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1561 2024-04-25T03:58:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1560 2024-04-25T03:58:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:19.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1562 2024-04-25T03:58:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1563 2024-04-25T03:58:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:20.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1565 2024-04-25T03:58:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:21.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1564 2024-04-25T03:58:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1568 2024-04-25T03:58:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1566 2024-04-25T03:58:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1567 2024-04-25T03:58:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:22.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1569 2024-04-25T03:58:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1571 2024-04-25T03:58:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1570 2024-04-25T03:58:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1572 2024-04-25T03:58:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1573 2024-04-25T03:58:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:24.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1575 2024-04-25T03:58:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1574 2024-04-25T03:58:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1577 2024-04-25T03:58:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1578 2024-04-25T03:58:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1576 2024-04-25T03:58:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1579 2024-04-25T03:58:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1580 2024-04-25T03:58:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1581 2024-04-25T03:58:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1583 2024-04-25T03:58:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1582 2024-04-25T03:58:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:30.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1585 2024-04-25T03:58:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1584 2024-04-25T03:58:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1588 2024-04-25T03:58:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:32.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1586 2024-04-25T03:58:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1587 2024-04-25T03:58:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1590 2024-04-25T03:58:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1589 2024-04-25T03:58:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1591 2024-04-25T03:58:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:33.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:33.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1593 2024-04-25T03:58:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1592 2024-04-25T03:58:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1595 2024-04-25T03:58:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1594 2024-04-25T03:58:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1598 2024-04-25T03:58:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1597 2024-04-25T03:58:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1596 2024-04-25T03:58:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:38.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1599 2024-04-25T03:58:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1600 2024-04-25T03:58:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1601 2024-04-25T03:58:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:38.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:38.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:39.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1602 2024-04-25T03:58:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1603 2024-04-25T03:58:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:40.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1605 2024-04-25T03:58:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1604 2024-04-25T03:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:41.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1608 2024-04-25T03:58:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1607 2024-04-25T03:58:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1606 2024-04-25T03:58:42.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1610 2024-04-25T03:58:42.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1609 2024-04-25T03:58:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1611 2024-04-25T03:58:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:42.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1613 2024-04-25T03:58:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1612 2024-04-25T03:58:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1615 2024-04-25T03:58:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T03:58:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T03:58:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T03:58:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T03:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T03:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T03:58:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1614 2024-04-25T03:58:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1617 2024-04-25T03:58:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1618 2024-04-25T03:58:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:47.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1616 2024-04-25T03:58:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T03:58:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T03:58:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T03:58:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1620 2024-04-25T03:58:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1619 2024-04-25T03:58:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1621 2024-04-25T03:58:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T03:58:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T03:58:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T03:58:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:48.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T03:58:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T03:58:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1622 2024-04-25T03:58:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1623 2024-04-25T03:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1625 2024-04-25T03:58:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1624 2024-04-25T03:58:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90144 2024-04-25T03:58:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90278 2024-04-25T03:58:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1628 2024-04-25T03:58:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T03:58:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T03:58:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T03:58:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1626 2024-04-25T03:58:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1627 2024-04-25T03:58:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1629 2024-04-25T03:58:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T03:58:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T03:58:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T03:58:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90412 2024-04-25T03:58:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T03:58:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T03:58:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T03:58:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1630 2024-04-25T03:58:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90566 2024-04-25T03:58:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T03:58:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T03:58:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T03:58:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T03:58:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T03:58:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T03:58:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90721 2024-04-25T03:58:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90876 2024-04-25T03:58:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 91031 2024-04-25T03:58:56.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1631 2024-04-25T03:58:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1632 2024-04-25T03:58:57.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T03:58:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 91206 2024-04-25T03:58:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T03:58:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 91382 2024-04-25T03:59:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T03:59:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T03:59:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90011 2024-04-25T04:01:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90145 2024-04-25T04:01:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90279 2024-04-25T04:01:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90413 2024-04-25T04:01:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90567 2024-04-25T04:01:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90722 2024-04-25T04:01:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 90877 2024-04-25T04:01:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 91032 2024-04-25T04:01:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:33.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:33.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 91207 2024-04-25T04:01:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:01:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 91383 2024-04-25T04:01:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:01:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:01:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:01:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T04:01:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T04:01:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T04:01:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T04:01:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T04:01:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T04:01:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T04:01:35.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:01:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:01:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T04:01:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T04:01:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T04:01:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T04:01:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T04:01:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T04:01:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T04:01:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T04:01:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:01:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T04:01:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:01:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97390 2024-04-25T04:01:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:01:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:01:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:01:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:01:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:01:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:01:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:01:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:01:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:01:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:01:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:02:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=02fea2e280d808517ace63aa2c82918c 2024-04-25T04:02:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T04:02:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T04:02:31.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T04:02:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=35c6a41d08a8cef51e2e92dbc067e637 2024-04-25T04:02:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T04:02:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T04:02:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T04:02:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=e1a7d05513ef7353286c0eb1bc727b88 2024-04-25T04:02:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T04:02:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T04:02:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T04:02:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=ea031f1214a1f5d68bed611b166641b9 2024-04-25T04:02:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T04:02:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T04:02:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=5676ca84c0900079a545f5d7d5cc6810 2024-04-25T04:02:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T04:02:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=e475bde581a1867413caf32a3e6ed9fd 2024-04-25T04:02:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T04:02:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T04:02:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:32.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T04:02:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=037ad135a8950a1c427c34dfcf43b230 2024-04-25T04:02:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T04:02:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T04:02:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T04:02:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=d6451326dd0964b317e8841583f39a96 2024-04-25T04:02:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T04:02:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T04:02:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T04:02:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T04:02:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T04:02:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=c92a7bc5900b67edf1924cd951b9276e 2024-04-25T04:02:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T04:02:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T04:02:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T04:02:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=85468081a16c985486b4a02b7c061867 2024-04-25T04:02:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T04:02:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T04:02:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T04:02:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T04:02:33.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T04:02:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:02:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:02:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T04:02:33.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T04:02:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T04:02:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T04:02:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T04:02:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T04:02:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:02:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1c188272-a2b9-4499-badd-37a62f0b1308 2024-04-25T04:02:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:02:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:02:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:02:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:02:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:02:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:02:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:02:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:02:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:02:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:02:34.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:02:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:02:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:02:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:34.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:02:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:02:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 252b1098-64c5-4869-8ed4-6442d5f44803 2024-04-25T04:02:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:02:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a9cb7225-1db7-4afc-8106-36f27b6ee1c3 2024-04-25T04:02:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:02:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd15e173-2ffd-404f-a02b-7b1a3d547f76 2024-04-25T04:02:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:34.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:34.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:02:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f3c2a3ad-06eb-4864-b13b-dcd75558a220 2024-04-25T04:02:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:02:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eab25d1c-b433-4102-92de-d533d0647beb 2024-04-25T04:02:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:02:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca 2024-04-25T04:02:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:02:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54b1f6ce-6058-4f9a-923a-4ecd1a629c84 2024-04-25T04:02:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:35.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:02:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5bb0fb4-fda6-43ff-9412-65e1c45bbb03 2024-04-25T04:02:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:02:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:02:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:02:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:02:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:02:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ffc8aa0e-7041-4b77-aba4-1b0cb879d619 2024-04-25T04:02:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:35.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:02:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:02:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1c188272-a2b9-4499-badd-37a62f0b1308 2024-04-25T04:02:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 252b1098-64c5-4869-8ed4-6442d5f44803 2024-04-25T04:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:02:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:02:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:02:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:02:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:02:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:02:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:02:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a9cb7225-1db7-4afc-8106-36f27b6ee1c3 2024-04-25T04:02:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:02:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:02:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:02:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:02:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:02:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f3c2a3ad-06eb-4864-b13b-dcd75558a220 2024-04-25T04:02:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:02:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd15e173-2ffd-404f-a02b-7b1a3d547f76 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:02:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eab25d1c-b433-4102-92de-d533d0647beb 2024-04-25T04:02:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:02:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54b1f6ce-6058-4f9a-923a-4ecd1a629c84 2024-04-25T04:02:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5bb0fb4-fda6-43ff-9412-65e1c45bbb03 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:02:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:02:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ffc8aa0e-7041-4b77-aba4-1b0cb879d619 2024-04-25T04:02:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:02:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:02:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1c188272-a2b9-4499-badd-37a62f0b1308 2024-04-25T04:02:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:39.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:02:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:40.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:02:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:02:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 252b1098-64c5-4869-8ed4-6442d5f44803 2024-04-25T04:02:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:02:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:02:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:02:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:02:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:02:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:02:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:02:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a9cb7225-1db7-4afc-8106-36f27b6ee1c3 2024-04-25T04:02:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:02:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:02:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f3c2a3ad-06eb-4864-b13b-dcd75558a220 2024-04-25T04:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd15e173-2ffd-404f-a02b-7b1a3d547f76 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:02:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eab25d1c-b433-4102-92de-d533d0647beb 2024-04-25T04:02:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:02:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54b1f6ce-6058-4f9a-923a-4ecd1a629c84 2024-04-25T04:02:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5bb0fb4-fda6-43ff-9412-65e1c45bbb03 2024-04-25T04:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:02:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:02:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ffc8aa0e-7041-4b77-aba4-1b0cb879d619 2024-04-25T04:02:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:02:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:02:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:16' 2024-04-25T04:02:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:02:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:02:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:02:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:02:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:02:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1c188272-a2b9-4499-badd-37a62f0b1308 2024-04-25T04:02:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:02:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:02:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:02:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:02:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 252b1098-64c5-4869-8ed4-6442d5f44803 2024-04-25T04:02:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:02:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T04:02:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T04:02:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:02:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:02:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:02:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:02:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:02:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:02:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:02:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a9cb7225-1db7-4afc-8106-36f27b6ee1c3 2024-04-25T04:02:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:02:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:02:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:02:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:02:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:02:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1c188272-a2b9-4499-badd-37a62f0b1308 2024-04-25T04:02:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:02:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:02:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f3c2a3ad-06eb-4864-b13b-dcd75558a220 2024-04-25T04:02:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:02:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca 2024-04-25T04:02:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd15e173-2ffd-404f-a02b-7b1a3d547f76 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:02:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eab25d1c-b433-4102-92de-d533d0647beb 2024-04-25T04:02:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5bb0fb4-fda6-43ff-9412-65e1c45bbb03 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:02:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54b1f6ce-6058-4f9a-923a-4ecd1a629c84 2024-04-25T04:02:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:02:49.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:02:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T04:02:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:02:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:02:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T04:02:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:02:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T04:02:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:02:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:02:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:02:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:02:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:02:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:02:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:02:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:02:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:02:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:02:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:49.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:02:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:02:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T04:02:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T04:02:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T04:02:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:02:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:02:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:02:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ffc8aa0e-7041-4b77-aba4-1b0cb879d619 2024-04-25T04:02:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:49.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:02:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:02:50.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:02:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:02:50.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:02:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T04:02:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T04:02:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:02:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:02:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 252b1098-64c5-4869-8ed4-6442d5f44803 2024-04-25T04:02:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:02:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T04:02:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T04:02:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:02:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T04:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:02:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T04:02:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:02:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a9cb7225-1db7-4afc-8106-36f27b6ee1c3 2024-04-25T04:02:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T04:02:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T04:02:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T04:02:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T04:02:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T04:02:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T04:02:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T04:02:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:02:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T04:02:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T04:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T04:02:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:02:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T04:02:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:02:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T04:02:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:02:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca 2024-04-25T04:02:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:02:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f3c2a3ad-06eb-4864-b13b-dcd75558a220 2024-04-25T04:02:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd15e173-2ffd-404f-a02b-7b1a3d547f76 2024-04-25T04:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:02:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eab25d1c-b433-4102-92de-d533d0647beb 2024-04-25T04:02:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54b1f6ce-6058-4f9a-923a-4ecd1a629c84 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:02:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5bb0fb4-fda6-43ff-9412-65e1c45bbb03 2024-04-25T04:02:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:50.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:02:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:51.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:02:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T04:02:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:02:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T04:02:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:02:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:02:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T04:02:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:02:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:51.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T04:02:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:02:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:02:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:02:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:02:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:02:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:02:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T04:02:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:02:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T04:02:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T04:02:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:02:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:02:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:02:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ffc8aa0e-7041-4b77-aba4-1b0cb879d619 2024-04-25T04:02:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:02:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:02:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:02:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:02:46' 2024-04-25T04:02:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:02:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:02:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T04:02:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:02:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T04:02:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T04:02:51.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:02:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:02:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:02:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T04:02:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:02:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T04:02:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:02:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:02:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:02:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:02:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 04:02:46' 2024-04-25T04:02:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:02:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:02:46' 2024-04-25T04:02:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:02:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:02:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:02:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:02:46' 2024-04-25T04:02:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:02:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:02:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T04:02:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T04:02:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:02:51.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:02:46' 2024-04-25T04:02:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:02:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:02:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:02:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:02:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:02:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:02:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:02:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T04:02:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:02:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:02:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:02:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:02:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T04:02:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:02:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T04:02:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T04:02:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T04:02:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T04:02:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T04:02:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T04:02:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:02:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:02:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T04:02:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T04:02:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:02:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T04:02:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:02:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T04:02:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:02:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T04:02:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T04:02:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:02:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:02:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T04:02:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:02:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:02:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T04:02:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:02:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:03:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=ea031f1214a1f5d68bed611b166641b9 2024-04-25T04:03:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T04:03:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=35c6a41d08a8cef51e2e92dbc067e637 2024-04-25T04:03:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T04:03:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 35c6a41d08a8cef51e2e92dbc067e637 != \3\5\c\6\a\4\1\d\0\8\a\8\c\e\f\5\1\e\2\e\9\2\d\b\c\0\6\7\e\6\3\7 ]] 2024-04-25T04:03:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ ea031f1214a1f5d68bed611b166641b9 != \e\a\0\3\1\f\1\2\1\4\a\1\f\5\d\6\8\b\e\d\6\1\1\b\1\6\6\6\4\1\b\9 ]] 2024-04-25T04:03:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=c92a7bc5900b67edf1924cd951b9276e 2024-04-25T04:03:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T04:03:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=02fea2e280d808517ace63aa2c82918c 2024-04-25T04:03:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T04:03:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=d6451326dd0964b317e8841583f39a96 2024-04-25T04:03:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T04:03:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ c92a7bc5900b67edf1924cd951b9276e != \c\9\2\a\7\b\c\5\9\0\0\b\6\7\e\d\f\1\9\2\4\c\d\9\5\1\b\9\2\7\6\e ]] 2024-04-25T04:03:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=5676ca84c0900079a545f5d7d5cc6810 2024-04-25T04:03:49.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T04:03:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 02fea2e280d808517ace63aa2c82918c != \0\2\f\e\a\2\e\2\8\0\d\8\0\8\5\1\7\a\c\e\6\3\a\a\2\c\8\2\9\1\8\c ]] 2024-04-25T04:03:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97391 2024-04-25T04:03:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ d6451326dd0964b317e8841583f39a96 != \d\6\4\5\1\3\2\6\d\d\0\9\6\4\b\3\1\7\e\8\8\4\1\5\8\3\f\3\9\a\9\6 ]] 2024-04-25T04:03:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97393 2024-04-25T04:03:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=e1a7d05513ef7353286c0eb1bc727b88 2024-04-25T04:03:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T04:03:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 5676ca84c0900079a545f5d7d5cc6810 != \5\6\7\6\c\a\8\4\c\0\9\0\0\0\7\9\a\5\4\5\f\5\d\7\d\5\c\c\6\8\1\0 ]] 2024-04-25T04:03:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ e1a7d05513ef7353286c0eb1bc727b88 != \e\1\a\7\d\0\5\5\1\3\e\f\7\3\5\3\2\8\6\c\0\e\b\1\b\c\7\2\7\b\8\8 ]] 2024-04-25T04:03:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=037ad135a8950a1c427c34dfcf43b230 2024-04-25T04:03:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T04:03:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 037ad135a8950a1c427c34dfcf43b230 != \0\3\7\a\d\1\3\5\a\8\9\5\0\a\1\c\4\2\7\c\3\4\d\f\c\f\4\3\b\2\3\0 ]] 2024-04-25T04:03:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=e475bde581a1867413caf32a3e6ed9fd 2024-04-25T04:03:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T04:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ e475bde581a1867413caf32a3e6ed9fd != \e\4\7\5\b\d\e\5\8\1\a\1\8\6\7\4\1\3\c\a\f\3\2\a\3\e\6\e\d\9\f\d ]] 2024-04-25T04:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97395 2024-04-25T04:03:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=85468081a16c985486b4a02b7c061867 2024-04-25T04:03:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 85468081a16c985486b4a02b7c061867 != \8\5\4\6\8\0\8\1\a\1\6\c\9\8\5\4\8\6\b\4\a\0\2\b\7\c\0\6\1\8\6\7 ]] 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97397 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97399 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97401 2024-04-25T04:03:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97403 2024-04-25T04:03:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97405 2024-04-25T04:03:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:03:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 97407 2024-04-25T04:03:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:03:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:03:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:03:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T04:03:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:03:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:03:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:03:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:03:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:03:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1c188272-a2b9-4499-badd-37a62f0b1308 2024-04-25T04:03:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:03:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3176652.8,"bytes_per_snapshot":63533056.0,"last_snapshot_bytes":63533056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017770,"remote_snapshot_timestamp":1714017770,"replay_state":"idle"} 2024-04-25T04:03:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:03:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:03:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:03:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:03:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:03:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3176652.8,"bytes_per_snapshot":63533056.0,"last_snapshot_bytes":63533056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017770,"remote_snapshot_timestamp":1714017770,"replay_state":"idle"} 2024-04-25T04:03:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:03:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:03:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:03:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:03:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:03:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:03:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:03:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:03:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:03:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:03:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:03:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:03:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:03:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:03:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:03:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary11c188272-a2b9-4499-badd-37a62f0b1308up+replayingreplaying, {"bytes_per_second":3176652.8,"bytes_per_snapshot":63533056.0,"last_snapshot_bytes":63533056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017770,"remote_snapshot_timestamp":1714017770,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+replayingreplaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+replayingreplaying, {"bytes_per_second":3602432.0,"bytes_per_snapshot":72048640.0,"last_snapshot_bytes":72048640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+replayingreplaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+replayingreplaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:03:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:03:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:03:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:03:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:03:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:03:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:03:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:03:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:03:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T04:03:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:03:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:03:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T04:03:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T04:03:52.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:03:52.355+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664481a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:03:52.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:03:52.356+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/1c188272-a2b9-4499-badd-37a62f0b1308] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:03:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:03:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T04:03:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:03:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:03:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:03:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T04:03:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T04:03:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:03:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:03:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T04:03:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:03:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:03:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:03:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:03:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:03:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:03:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:03:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 54b1f6ce-6058-4f9a-923a-4ecd1a629c84 2024-04-25T04:03:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:03:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:03:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:03:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:03:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:03:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:03:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:03:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:03:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:03:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:03:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:03:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:03:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:03:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:03:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:03:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:03:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:03:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:03:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:03:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary254b1f6ce-6058-4f9a-923a-4ecd1a629c84up+replayingreplaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+replayingreplaying, {"bytes_per_second":3602432.0,"bytes_per_snapshot":72048640.0,"last_snapshot_bytes":72048640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+replayingreplaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+replayingreplaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:03:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:03:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:03:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:03:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:03:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:03:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:03:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:03:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:03:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T04:03:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:03:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:03:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T04:03:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T04:03:56.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:03:56.180+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667bc6ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:03:56.185 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:03:56.180+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508a00 [3/54b1f6ce-6058-4f9a-923a-4ecd1a629c84] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:03:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:03:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T04:03:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:03:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:03:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:03:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T04:03:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T04:03:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:03:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:03:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:03:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:03:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:03:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:03:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:03:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: c5bb0fb4-fda6-43ff-9412-65e1c45bbb03 2024-04-25T04:03:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:03:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:03:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:03:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:03:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:03:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:03:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:03:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:03:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:03:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:03:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:03:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:03:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:03:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:03:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:03:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:03:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:03:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:03:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:03:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary3c5bb0fb4-fda6-43ff-9412-65e1c45bbb03up+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+replayingreplaying, {"bytes_per_second":3602432.0,"bytes_per_snapshot":72048640.0,"last_snapshot_bytes":72048640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+replayingreplaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+replayingreplaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:03:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:03:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:03:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:03:58.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:03:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:03:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:03:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:03:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:03:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:03:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T04:03:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:03:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:03:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:03:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T04:03:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T04:03:58.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:03:58.917+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966845e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:03:58.922 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:03:58.917+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/c5bb0fb4-fda6-43ff-9412-65e1c45bbb03] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:00.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T04:04:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T04:04:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T04:04:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:04:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T04:04:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:04:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:04:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca 2024-04-25T04:04:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:04:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:04:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:04:01.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:04:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:04:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:04:01.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:04:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:04:01.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:04:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:04:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:04:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:04:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary42fd7a669-72b1-4c8b-8bb6-3013d96ad7caup+replayingreplaying, {"bytes_per_second":3609600.0,"bytes_per_snapshot":72192000.0,"last_snapshot_bytes":72192000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+replayingreplaying, {"bytes_per_second":3602432.0,"bytes_per_snapshot":72048640.0,"last_snapshot_bytes":72048640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+replayingreplaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+replayingreplaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:04:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:04:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:04:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:04:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:04:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:04:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T04:04:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:04:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T04:04:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T04:04:01.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:01.905+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664790820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:01.909 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:01.905+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/2fd7a669-72b1-4c8b-8bb6-3013d96ad7ca] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:03.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T04:04:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T04:04:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T04:04:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:04:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:04:04.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:04:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: eab25d1c-b433-4102-92de-d533d0647beb 2024-04-25T04:04:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:04:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3602432.0,"bytes_per_snapshot":72048640.0,"last_snapshot_bytes":72048640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:04:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:04:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:04:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:04:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:04:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3602432.0,"bytes_per_snapshot":72048640.0,"last_snapshot_bytes":72048640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:04:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:04.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:04:04.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:04.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:04:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:04:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:04:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:04:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:04:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary5eab25d1c-b433-4102-92de-d533d0647bebup+replayingreplaying, {"bytes_per_second":3602432.0,"bytes_per_snapshot":72048640.0,"last_snapshot_bytes":72048640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+replayingreplaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+replayingreplaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:04:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:04:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:04:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:04:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:04:04.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:04:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T04:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T04:04:04.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T04:04:04.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:04.791+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596652f7d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:04.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:04.791+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/eab25d1c-b433-4102-92de-d533d0647beb] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T04:04:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T04:04:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T04:04:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:04:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T04:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:07.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:04:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:04:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a9cb7225-1db7-4afc-8106-36f27b6ee1c3 2024-04-25T04:04:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:04:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"} 2024-04-25T04:04:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:04:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:04:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:04:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:04:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:04:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"} 2024-04-25T04:04:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:04:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:04:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:04:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:04:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:04:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:04:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:04:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary6a9cb7225-1db7-4afc-8106-36f27b6ee1c3up+replayingreplaying, {"bytes_per_second":3649126.4,"bytes_per_snapshot":72982528.0,"last_snapshot_bytes":72982528,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+replayingreplaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:04:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:04:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:04:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:04:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:04:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:04:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T04:04:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:04:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T04:04:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T04:04:07.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:07.823+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667879380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:07.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:07.823+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/a9cb7225-1db7-4afc-8106-36f27b6ee1c3] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T04:04:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:09.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T04:04:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:04:09.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:04:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T04:04:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 252b1098-64c5-4869-8ed4-6442d5f44803 2024-04-25T04:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:04:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"} 2024-04-25T04:04:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:04:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:04:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:04:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:04:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:04:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"} 2024-04-25T04:04:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:04:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:04:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:04:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:04:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:04:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:04:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:04:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary7252b1098-64c5-4869-8ed4-6442d5f44803up+replayingreplaying, {"bytes_per_second":3672678.4,"bytes_per_snapshot":73453568.0,"last_snapshot_bytes":73453568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017771,"remote_snapshot_timestamp":1714017771,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:04:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:04:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:04:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:04:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:04:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:04:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T04:04:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:04:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T04:04:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T04:04:10.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:10.812+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668498d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:10.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:10.812+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/252b1098-64c5-4869-8ed4-6442d5f44803] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:12.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T04:04:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:12.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T04:04:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T04:04:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:04:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T04:04:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:04:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:04:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ffc8aa0e-7041-4b77-aba4-1b0cb879d619 2024-04-25T04:04:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:04:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:04:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:04:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:04:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:04:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:04:13.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:04:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:04:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:04:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:04:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:04:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:04:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:04:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary8ffc8aa0e-7041-4b77-aba4-1b0cb879d619up+replayingreplaying, {"bytes_per_second":3608780.8,"bytes_per_snapshot":72175616.0,"last_snapshot_bytes":72175616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:04:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:04:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:04:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:04:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:04:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:04:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T04:04:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:04:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T04:04:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T04:04:13.803 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:13.799+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667a4eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:13.804 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:13.799+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508780 [3/ffc8aa0e-7041-4b77-aba4-1b0cb879d619] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T04:04:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T04:04:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:15.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:04:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T04:04:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:04:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:04:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f3c2a3ad-06eb-4864-b13b-dcd75558a220 2024-04-25T04:04:16.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:04:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:04:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:03:46' 2024-04-25T04:04:16.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:04:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:04:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:04:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:16.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:04:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:04:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:16.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:04:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:04:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:04:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:04:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 04:03:46' 2024-04-25T04:04:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:03:46' 2024-04-25T04:04:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:04:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":3596902.4,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46image-primary9f3c2a3ad-06eb-4864-b13b-dcd75558a220up+replayingreplaying, {"bytes_per_second":3571507.2,"bytes_per_snapshot":71430144.0,"last_snapshot_bytes":71430144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:03:46' 2024-04-25T04:04:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:04:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:03:46' 2024-04-25T04:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:04:16.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:04:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:16.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:04:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T04:04:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:04:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T04:04:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T04:04:16.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:16.806+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966845e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:16.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:16.806+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/f3c2a3ad-06eb-4864-b13b-dcd75558a220] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:16.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:16.833+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664f05a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:16.838 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:16.833+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/f3c2a3ad-06eb-4864-b13b-dcd75558a220] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T04:04:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:18.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:18.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T04:04:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T04:04:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:04:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T04:04:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:04:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:04:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:04:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:04:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd15e173-2ffd-404f-a02b-7b1a3d547f76 2024-04-25T04:04:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:04:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:04:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:04:16' 2024-04-25T04:04:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:04:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:04:19.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:04:19.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"} 2024-04-25T04:04:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:04:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:19.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:04:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:04:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:04:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:04:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:04:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:04:16' 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 04:04:16' 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:04:16' 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:04:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:04:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10fd15e173-2ffd-404f-a02b-7b1a3d547f76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71938048.0,"last_snapshot_bytes":71938048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714017772,"remote_snapshot_timestamp":1714017772,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:04:16' 2024-04-25T04:04:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:04:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:04:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:04:16' 2024-04-25T04:04:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:04:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:04:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:04:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:04:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:04:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T04:04:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:04:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T04:04:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T04:04:19.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:19.803+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c63d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:04:19.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:04:19.803+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508c80 [3/fd15e173-2ffd-404f-a02b-7b1a3d547f76] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:04:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:04:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T04:04:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:04:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:04:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T04:04:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T04:04:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T04:04:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T04:04:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T04:04:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T04:04:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:21.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:21.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T04:04:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T04:04:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T04:04:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 294472c834c7 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.294472c834c7 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:21 2024 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:21 2024 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:21 2024 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 68825e16-9975-4692-b7f2-c9e1c17fdb0d 2024-04-25T04:04:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T04:04:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T04:04:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T04:04:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ef023057-c1d4-41e9-8e1b-82bb6e48481b 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T04:04:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T04:04:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T04:04:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T04:04:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T04:04:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T04:04:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T04:04:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T04:04:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1654 2024-04-25T04:04:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 295455b2e579 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.295455b2e579 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:22 2024 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:22 2024 2024-04-25T04:04:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:22 2024 2024-04-25T04:04:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: bae18c92-73f9-406c-a794-546ed67a522b 2024-04-25T04:04:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T04:04:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T04:04:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T04:04:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 7179c7b5-063b-4ebe-b31c-1cc6cf50c87f 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:23.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T04:04:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T04:04:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T04:04:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T04:04:23.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:04:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:23.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T04:04:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T04:04:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1656 2024-04-25T04:04:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T04:04:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T04:04:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 29636b4c67d0 2024-04-25T04:04:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.29636b4c67d0 2024-04-25T04:04:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:23 2024 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:23 2024 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:23 2024 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: d848cdb9-de5e-4504-9a7c-224a846e7c5e 2024-04-25T04:04:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T04:04:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T04:04:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T04:04:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 0754fada-e19f-4ed8-a899-0ec4c5f3ee5e 2024-04-25T04:04:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T04:04:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T04:04:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T04:04:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T04:04:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T04:04:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T04:04:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T04:04:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1658 2024-04-25T04:04:25.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T04:04:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T04:04:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2972f8533937 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2972f8533937 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:24 2024 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:24 2024 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:24 2024 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:04:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:25.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T04:04:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T04:04:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T04:04:25.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 2d4702a3-5834-4665-bc8f-2f6390b79b7b 2024-04-25T04:04:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T04:04:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T04:04:25.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:04:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T04:04:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T04:04:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T04:04:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T04:04:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T04:04:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T04:04:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T04:04:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1660 2024-04-25T04:04:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T04:04:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T04:04:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2981af355df1 2024-04-25T04:04:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2981af355df1 2024-04-25T04:04:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:25 2024 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:25 2024 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:25 2024 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 00618b50-8806-4f74-87ae-e83a03ea5580 2024-04-25T04:04:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:26.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T04:04:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T04:04:26.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T04:04:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 6d229d19-6ab9-4911-9dfd-a133053d720d 2024-04-25T04:04:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T04:04:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T04:04:26.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T04:04:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T04:04:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:27.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T04:04:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T04:04:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:27.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:27.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1662 2024-04-25T04:04:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1663 2024-04-25T04:04:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T04:04:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 29916867de2e 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.29916867de2e 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:27 2024 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:27 2024 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:27 2024 2024-04-25T04:04:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: a70e20c2-c6c2-4ab7-83af-dacabb40573f 2024-04-25T04:04:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T04:04:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T04:04:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T04:04:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: b15ceef7-1811-4269-b47d-904dc4aedad2 2024-04-25T04:04:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T04:04:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T04:04:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T04:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T04:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T04:04:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T04:04:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T04:04:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:28.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T04:04:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1665 2024-04-25T04:04:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1668 2024-04-25T04:04:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1666 2024-04-25T04:04:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T04:04:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 29a3785430f7 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.29a3785430f7 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:28 2024 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:28 2024 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:28 2024 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 626c9677-cb63-430c-925d-37f989447e3b 2024-04-25T04:04:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T04:04:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T04:04:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T04:04:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 0f0a53ef-1f75-4a8c-b29e-b261b1ab75a1 2024-04-25T04:04:29.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:29.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T04:04:29.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T04:04:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T04:04:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T04:04:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T04:04:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T04:04:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T04:04:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1670 2024-04-25T04:04:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1669 2024-04-25T04:04:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T04:04:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:30.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:30.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 29b6babb75df 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.29b6babb75df 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:29 2024 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:29 2024 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:29 2024 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:04:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T04:04:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T04:04:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T04:04:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 607552f0-21e1-4a06-95a7-e9855d5c5ed5 2024-04-25T04:04:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T04:04:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T04:04:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T04:04:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T04:04:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:04:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:04:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:04:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T04:04:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T04:04:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T04:04:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T04:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T04:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T04:04:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T04:04:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T04:04:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T04:04:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T04:04:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1673 2024-04-25T04:04:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1674 2024-04-25T04:04:31.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1672 2024-04-25T04:04:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T04:04:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T04:04:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 29ca586d9a53 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.29ca586d9a53 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:30 2024 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:30 2024 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:30 2024 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d 2024-04-25T04:04:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T04:04:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T04:04:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T04:04:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:31.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 5a9bd545-5c09-4db1-bcff-b9e483f240d5 2024-04-25T04:04:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:31.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T04:04:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T04:04:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:04:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:04:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T04:04:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T04:04:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1677 2024-04-25T04:04:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1676 2024-04-25T04:04:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:32.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:04:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:32.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T04:04:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T04:04:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:04:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:04:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:04:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 29dcbec9cdc 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.29dcbec9cdc 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:04:31 2024 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:04:31 2024 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:04:31 2024 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:04:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:04:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:04:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:04:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T04:04:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T04:04:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:04:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T04:04:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 52c5d525-a349-4cad-9d5f-6f878616b0e0 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:04:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:04:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:04:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:04:32.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:04:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T04:04:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T04:04:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:04:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T04:04:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:04:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:04:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:04:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 101837 2024-04-25T04:04:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T04:04:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T04:04:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T04:04:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:04:33.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T04:04:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1679 2024-04-25T04:04:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1680 2024-04-25T04:04:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1681 2024-04-25T04:04:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:04:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:33.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:04:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:04:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1682 2024-04-25T04:04:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1683 2024-04-25T04:04:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:04:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:04:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1684 2024-04-25T04:04:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1686 2024-04-25T04:04:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1685 2024-04-25T04:04:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:04:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1687 2024-04-25T04:04:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1688 2024-04-25T04:04:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1691 2024-04-25T04:04:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1690 2024-04-25T04:04:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1689 2024-04-25T04:04:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:04:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:04:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1692 2024-04-25T04:04:38.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1693 2024-04-25T04:04:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1694 2024-04-25T04:04:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1695 2024-04-25T04:04:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1696 2024-04-25T04:04:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:04:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:04:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:04:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:04:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1698 2024-04-25T04:04:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1697 2024-04-25T04:04:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:04:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1701 2024-04-25T04:04:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1699 2024-04-25T04:04:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1700 2024-04-25T04:04:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:41.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:04:42.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1702 2024-04-25T04:04:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1703 2024-04-25T04:04:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1705 2024-04-25T04:04:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1706 2024-04-25T04:04:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:04:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:04:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1708 2024-04-25T04:04:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:04:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:04:44.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:45.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1704 2024-04-25T04:04:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1709 2024-04-25T04:04:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1707 2024-04-25T04:04:45.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1710 2024-04-25T04:04:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:04:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:04:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:46.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1711 2024-04-25T04:04:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1712 2024-04-25T04:04:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:04:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:04:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1715 2024-04-25T04:04:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1713 2024-04-25T04:04:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1714 2024-04-25T04:04:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1716 2024-04-25T04:04:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:04:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:48.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:48.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1719 2024-04-25T04:04:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1717 2024-04-25T04:04:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:04:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1720 2024-04-25T04:04:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:04:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:04:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1718 2024-04-25T04:04:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1721 2024-04-25T04:04:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1722 2024-04-25T04:04:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:04:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:04:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:04:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:04:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1725 2024-04-25T04:04:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1723 2024-04-25T04:04:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1726 2024-04-25T04:04:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1724 2024-04-25T04:04:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:53.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:53.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1727 2024-04-25T04:04:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1728 2024-04-25T04:04:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:04:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1729 2024-04-25T04:04:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1730 2024-04-25T04:04:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:04:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:04:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:04:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:04:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:04:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:04:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:04:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:04:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:04:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:04:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:04:56.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:04:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1731 2024-04-25T04:04:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1732 2024-04-25T04:04:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1734 2024-04-25T04:04:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:04:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:04:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:04:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:04:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:04:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:04:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1733 2024-04-25T04:04:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:04:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:04:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:04:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:04:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:04:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:04:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1735 2024-04-25T04:04:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1736 2024-04-25T04:04:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:04:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:04:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:04:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:04:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:04:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:04:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:04:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:04:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:04:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1738 2024-04-25T04:05:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1737 2024-04-25T04:05:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1739 2024-04-25T04:05:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1740 2024-04-25T04:05:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1742 2024-04-25T04:05:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1741 2024-04-25T04:05:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:02.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1744 2024-04-25T04:05:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1743 2024-04-25T04:05:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:03.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1745 2024-04-25T04:05:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1746 2024-04-25T04:05:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1747 2024-04-25T04:05:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1748 2024-04-25T04:05:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1749 2024-04-25T04:05:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1750 2024-04-25T04:05:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1752 2024-04-25T04:05:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1751 2024-04-25T04:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:06.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1754 2024-04-25T04:05:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1753 2024-04-25T04:05:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:08.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1755 2024-04-25T04:05:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1756 2024-04-25T04:05:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1757 2024-04-25T04:05:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1758 2024-04-25T04:05:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:11.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1760 2024-04-25T04:05:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1759 2024-04-25T04:05:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:11.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1762 2024-04-25T04:05:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1761 2024-04-25T04:05:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1764 2024-04-25T04:05:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1763 2024-04-25T04:05:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1766 2024-04-25T04:05:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1765 2024-04-25T04:05:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:14.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1767 2024-04-25T04:05:14.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1768 2024-04-25T04:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1770 2024-04-25T04:05:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1769 2024-04-25T04:05:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:15.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:15.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1772 2024-04-25T04:05:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:16.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1771 2024-04-25T04:05:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:17.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1774 2024-04-25T04:05:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1773 2024-04-25T04:05:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:19.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1775 2024-04-25T04:05:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1776 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1778 2024-04-25T04:05:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1777 2024-04-25T04:05:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1780 2024-04-25T04:05:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1779 2024-04-25T04:05:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:21.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:21.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:05:21.475+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667961800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:21.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:05:21.487+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667960800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:05:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1781 2024-04-25T04:05:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1782 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1784 2024-04-25T04:05:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1783 2024-04-25T04:05:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1785 2024-04-25T04:05:23.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1786 2024-04-25T04:05:23.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1788 2024-04-25T04:05:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1787 2024-04-25T04:05:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1789 2024-04-25T04:05:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1790 2024-04-25T04:05:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:26.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:26.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:26.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1791 2024-04-25T04:05:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1792 2024-04-25T04:05:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1794 2024-04-25T04:05:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1793 2024-04-25T04:05:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1795 2024-04-25T04:05:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1796 2024-04-25T04:05:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1798 2024-04-25T04:05:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1797 2024-04-25T04:05:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1801 2024-04-25T04:05:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1802 2024-04-25T04:05:30.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1800 2024-04-25T04:05:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1799 2024-04-25T04:05:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1804 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1803 2024-04-25T04:05:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1806 2024-04-25T04:05:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1805 2024-04-25T04:05:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1807 2024-04-25T04:05:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1810 2024-04-25T04:05:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1809 2024-04-25T04:05:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1808 2024-04-25T04:05:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1811 2024-04-25T04:05:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1812 2024-04-25T04:05:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:36.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:36.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1814 2024-04-25T04:05:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1813 2024-04-25T04:05:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1819 2024-04-25T04:05:38.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1818 2024-04-25T04:05:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1815 2024-04-25T04:05:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1816 2024-04-25T04:05:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1817 2024-04-25T04:05:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1820 2024-04-25T04:05:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1822 2024-04-25T04:05:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1821 2024-04-25T04:05:41.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:41.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:42.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1823 2024-04-25T04:05:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:42.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1824 2024-04-25T04:05:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1825 2024-04-25T04:05:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1826 2024-04-25T04:05:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1828 2024-04-25T04:05:43.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1827 2024-04-25T04:05:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1829 2024-04-25T04:05:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1830 2024-04-25T04:05:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1831 2024-04-25T04:05:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1832 2024-04-25T04:05:45.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:45.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:46.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:46.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1833 2024-04-25T04:05:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1836 2024-04-25T04:05:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1835 2024-04-25T04:05:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1834 2024-04-25T04:05:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1837 2024-04-25T04:05:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1838 2024-04-25T04:05:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1840 2024-04-25T04:05:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1839 2024-04-25T04:05:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1841 2024-04-25T04:05:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1842 2024-04-25T04:05:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1843 2024-04-25T04:05:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1846 2024-04-25T04:05:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1848 2024-04-25T04:05:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1847 2024-04-25T04:05:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1844 2024-04-25T04:05:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1845 2024-04-25T04:05:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1850 2024-04-25T04:05:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1849 2024-04-25T04:05:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1851 2024-04-25T04:05:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1852 2024-04-25T04:05:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:05:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:05:55.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:05:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1853 2024-04-25T04:05:56.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:05:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:05:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:05:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:05:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:05:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:05:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:05:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:05:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:05:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:05:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:05:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:05:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:05:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:05:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:05:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:05:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:05:57.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:05:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:05:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:05:58.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:05:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:05:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:05:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:05:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1854 2024-04-25T04:05:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1855 2024-04-25T04:05:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1856 2024-04-25T04:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:05:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:05:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1860 2024-04-25T04:05:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1859 2024-04-25T04:05:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:05:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1861 2024-04-25T04:06:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1862 2024-04-25T04:06:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:01.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1857 2024-04-25T04:06:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1858 2024-04-25T04:06:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1863 2024-04-25T04:06:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:01.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1864 2024-04-25T04:06:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:02.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:02.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1865 2024-04-25T04:06:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1866 2024-04-25T04:06:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1867 2024-04-25T04:06:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1868 2024-04-25T04:06:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:05.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1870 2024-04-25T04:06:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1869 2024-04-25T04:06:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1871 2024-04-25T04:06:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1872 2024-04-25T04:06:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1873 2024-04-25T04:06:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1874 2024-04-25T04:06:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1876 2024-04-25T04:06:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1875 2024-04-25T04:06:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1877 2024-04-25T04:06:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1878 2024-04-25T04:06:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1881 2024-04-25T04:06:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1880 2024-04-25T04:06:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1882 2024-04-25T04:06:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1879 2024-04-25T04:06:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1883 2024-04-25T04:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1884 2024-04-25T04:06:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:12.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1885 2024-04-25T04:06:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1886 2024-04-25T04:06:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:13.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:13.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:13.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1888 2024-04-25T04:06:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1887 2024-04-25T04:06:14.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:14.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1892 2024-04-25T04:06:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1890 2024-04-25T04:06:15.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1889 2024-04-25T04:06:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1891 2024-04-25T04:06:15.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1893 2024-04-25T04:06:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1894 2024-04-25T04:06:18.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:06:18.295+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664cd4000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:06:18.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:06:18.309+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667a75000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:06:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1895 2024-04-25T04:06:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1896 2024-04-25T04:06:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:19.316 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:06:19.311+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667576c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:06:19.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:06:19.332+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596659aa400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:06:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1897 2024-04-25T04:06:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1898 2024-04-25T04:06:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:20.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:06:20.350+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966777a800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:06:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1902 2024-04-25T04:06:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1900 2024-04-25T04:06:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1901 2024-04-25T04:06:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1899 2024-04-25T04:06:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:22.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1903 2024-04-25T04:06:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1904 2024-04-25T04:06:22.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:23.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1905 2024-04-25T04:06:23.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1906 2024-04-25T04:06:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:23.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1907 2024-04-25T04:06:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1908 2024-04-25T04:06:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:24.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1912 2024-04-25T04:06:25.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1910 2024-04-25T04:06:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1911 2024-04-25T04:06:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1909 2024-04-25T04:06:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1913 2024-04-25T04:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:27.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:27.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1916 2024-04-25T04:06:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1915 2024-04-25T04:06:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1914 2024-04-25T04:06:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1918 2024-04-25T04:06:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1917 2024-04-25T04:06:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:28.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1919 2024-04-25T04:06:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1921 2024-04-25T04:06:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1920 2024-04-25T04:06:31.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1922 2024-04-25T04:06:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1924 2024-04-25T04:06:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1925 2024-04-25T04:06:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1923 2024-04-25T04:06:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1926 2024-04-25T04:06:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1927 2024-04-25T04:06:32.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1928 2024-04-25T04:06:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1929 2024-04-25T04:06:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:35.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1932 2024-04-25T04:06:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1933 2024-04-25T04:06:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1930 2024-04-25T04:06:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1931 2024-04-25T04:06:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1934 2024-04-25T04:06:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1935 2024-04-25T04:06:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1938 2024-04-25T04:06:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1936 2024-04-25T04:06:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:36.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1939 2024-04-25T04:06:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1937 2024-04-25T04:06:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:38.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:39.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:39.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:06:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:06:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:06:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1942 2024-04-25T04:06:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1941 2024-04-25T04:06:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:41.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1946 2024-04-25T04:06:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1940 2024-04-25T04:06:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1947 2024-04-25T04:06:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1944 2024-04-25T04:06:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1943 2024-04-25T04:06:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:41.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1949 2024-04-25T04:06:41.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:42.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1945 2024-04-25T04:06:42.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1948 2024-04-25T04:06:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:06:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:06:43.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:06:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:43.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 101971 2024-04-25T04:06:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:06:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:06:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:06:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:44.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:06:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:06:44.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:06:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:06:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:06:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:06:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1950 2024-04-25T04:06:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1951 2024-04-25T04:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:06:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:06:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:06:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:45.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1955 2024-04-25T04:06:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1953 2024-04-25T04:06:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1954 2024-04-25T04:06:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1952 2024-04-25T04:06:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1957 2024-04-25T04:06:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1956 2024-04-25T04:06:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1958 2024-04-25T04:06:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:06:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:06:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:06:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:06:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:06:48.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:06:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102105 2024-04-25T04:06:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102239 2024-04-25T04:06:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:06:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:06:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:06:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:06:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:06:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:06:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:06:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:06:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:06:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102393 2024-04-25T04:06:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102548 2024-04-25T04:06:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1960 2024-04-25T04:06:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1959 2024-04-25T04:06:51.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1962 2024-04-25T04:06:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1961 2024-04-25T04:06:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:06:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102724 2024-04-25T04:06:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102899 2024-04-25T04:06:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 103055 2024-04-25T04:06:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:06:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 103230 2024-04-25T04:06:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:06:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:06:56.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 101838 2024-04-25T04:09:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 101972 2024-04-25T04:09:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102106 2024-04-25T04:09:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102240 2024-04-25T04:09:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102394 2024-04-25T04:09:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102549 2024-04-25T04:09:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102725 2024-04-25T04:09:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 102900 2024-04-25T04:09:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 103056 2024-04-25T04:09:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:09:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 103231 2024-04-25T04:09:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:09:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:09:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:09:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T04:09:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T04:09:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T04:09:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T04:09:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T04:09:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T04:09:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T04:09:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T04:09:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:09:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109202 2024-04-25T04:09:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T04:09:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T04:09:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T04:09:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T04:09:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:09:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T04:09:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:09:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:33.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:09:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:09:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:09:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:09:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:09:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:09:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:09:34.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:09:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:09:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:10:26.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=720334046a482935d8a93060cc0b65e5 2024-04-25T04:10:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T04:10:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T04:10:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T04:10:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=e0852dd5662ea2879d7dc8d5a7ccfbde 2024-04-25T04:10:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T04:10:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T04:10:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T04:10:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=df2cadf3b682e1cd6a30cec5ceaecd15 2024-04-25T04:10:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T04:10:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T04:10:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T04:10:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T04:10:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:29.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=724f6ab49b347a1d6184eed73b347d7c 2024-04-25T04:10:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T04:10:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T04:10:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T04:10:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=a862ed835a4d735555eaa860e874f3b6 2024-04-25T04:10:29.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T04:10:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T04:10:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T04:10:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:10:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:10:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:10:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:10:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=aa6cd30be68971db17cf70ef2e0ecb5f 2024-04-25T04:10:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T04:10:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=bda3e095eac904089de98cb16679a067 2024-04-25T04:10:30.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T04:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:10:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:10:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:10:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:10:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=c01110ed2409da0e46966ec539c780f1 2024-04-25T04:10:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T04:10:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T04:10:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T04:10:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T04:10:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T04:10:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T04:10:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T04:10:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=a976023555e0094dbf6337d02f65a1ba 2024-04-25T04:10:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T04:10:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=94724bf3f56387884d469b3f184d9daa 2024-04-25T04:10:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T04:10:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T04:10:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T04:10:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T04:10:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T04:10:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T04:10:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T04:10:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:10:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:10:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T04:10:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:10:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a70e20c2-c6c2-4ab7-83af-dacabb40573f 2024-04-25T04:10:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:32.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:10:32.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:32.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:10:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:10:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:10:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:10:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:10:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:10:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:10:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:33.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:10:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:10:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:33.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:10:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:10:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:10:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:10:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bae18c92-73f9-406c-a794-546ed67a522b 2024-04-25T04:10:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:10:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d848cdb9-de5e-4504-9a7c-224a846e7c5e 2024-04-25T04:10:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:10:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:10:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d 2024-04-25T04:10:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:10:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:10:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:10:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a70e20c2-c6c2-4ab7-83af-dacabb40573f 2024-04-25T04:10:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:10:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T04:10:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:10:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:10:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 626c9677-cb63-430c-925d-37f989447e3b 2024-04-25T04:10:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:10:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:10:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:10:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:10:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:10:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bae18c92-73f9-406c-a794-546ed67a522b 2024-04-25T04:10:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:10:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d848cdb9-de5e-4504-9a7c-224a846e7c5e 2024-04-25T04:10:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:10:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:10:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:10:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:10:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:10:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:10:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 68825e16-9975-4692-b7f2-c9e1c17fdb0d 2024-04-25T04:10:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:10:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 00618b50-8806-4f74-87ae-e83a03ea5580 2024-04-25T04:10:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:10:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:10:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d 2024-04-25T04:10:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:10:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:10:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:10:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:10:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:10:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:10:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:10:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:10:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:10:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:37.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:37.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:10:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:10:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:10:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 626c9677-cb63-430c-925d-37f989447e3b 2024-04-25T04:10:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:10:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:10:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:10:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 00618b50-8806-4f74-87ae-e83a03ea5580 2024-04-25T04:10:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:38.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:10:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 68825e16-9975-4692-b7f2-c9e1c17fdb0d 2024-04-25T04:10:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:10:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:10:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:10:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:10:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a70e20c2-c6c2-4ab7-83af-dacabb40573f 2024-04-25T04:10:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:10:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:10:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:10:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:10:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:10:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bae18c92-73f9-406c-a794-546ed67a522b 2024-04-25T04:10:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d848cdb9-de5e-4504-9a7c-224a846e7c5e 2024-04-25T04:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:10:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:40.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:10:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:10:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d 2024-04-25T04:10:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:10:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:10:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:10:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 626c9677-cb63-430c-925d-37f989447e3b 2024-04-25T04:10:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:10:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:10:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:10:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:10:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 00618b50-8806-4f74-87ae-e83a03ea5580 2024-04-25T04:10:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:10:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 68825e16-9975-4692-b7f2-c9e1c17fdb0d 2024-04-25T04:10:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:10:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:45.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:10:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:10:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:10:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:10:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:10:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:45.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:10:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:10:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:10:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:10:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:10:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:10:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:16' 2024-04-25T04:10:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:10:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:10:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:10:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:10:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:10:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a70e20c2-c6c2-4ab7-83af-dacabb40573f 2024-04-25T04:10:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:47.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:10:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:10:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:47.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:10:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:10:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:10:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:10:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:10:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bae18c92-73f9-406c-a794-546ed67a522b 2024-04-25T04:10:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:47.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:10:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d848cdb9-de5e-4504-9a7c-224a846e7c5e 2024-04-25T04:10:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:47.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:10:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:10:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T04:10:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:48.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:10:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:10:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:48.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T04:10:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T04:10:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:10:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a70e20c2-c6c2-4ab7-83af-dacabb40573f 2024-04-25T04:10:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:10:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:10:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T04:10:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:10:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:10:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:10:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d 2024-04-25T04:10:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:48.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:48.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:10:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:10:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T04:10:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T04:10:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T04:10:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:10:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bae18c92-73f9-406c-a794-546ed67a522b 2024-04-25T04:10:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:10:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d848cdb9-de5e-4504-9a7c-224a846e7c5e 2024-04-25T04:10:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:10:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:10:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T04:10:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T04:10:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:10:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:10:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T04:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T04:10:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T04:10:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:10:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:10:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:10:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 626c9677-cb63-430c-925d-37f989447e3b 2024-04-25T04:10:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T04:10:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T04:10:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:10:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:10:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:50.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T04:10:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:10:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d 2024-04-25T04:10:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:10:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:10:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T04:10:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:10:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T04:10:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T04:10:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T04:10:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T04:10:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T04:10:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:10:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:10:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:50.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 00618b50-8806-4f74-87ae-e83a03ea5580 2024-04-25T04:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:50.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:50.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:10:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 68825e16-9975-4692-b7f2-c9e1c17fdb0d 2024-04-25T04:10:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:50.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T04:10:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T04:10:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:10:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:10:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T04:10:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:10:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:10:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T04:10:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:10:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 626c9677-cb63-430c-925d-37f989447e3b 2024-04-25T04:10:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:51.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:10:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:10:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:51.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T04:10:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:10:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T04:10:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T04:10:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T04:10:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:10:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T04:10:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T04:10:52.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T04:10:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:10:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 00618b50-8806-4f74-87ae-e83a03ea5580 2024-04-25T04:10:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:10:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 68825e16-9975-4692-b7f2-c9e1c17fdb0d 2024-04-25T04:10:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:52.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T04:10:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:10:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:10:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T04:10:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:10:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T04:10:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:10:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:10:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T04:10:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T04:10:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:10:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:10:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:10:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:10:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:10:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:10:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:10:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:53.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T04:10:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T04:10:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:10:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:10:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:10:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:10:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:53.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:53.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:53.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T04:10:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:10:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T04:10:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:10:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:10:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:10:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:10:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:53.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:10:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T04:10:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:53.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:10:53.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:10:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:53.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:10:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T04:10:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T04:10:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T04:10:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T04:10:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:10:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:10:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:54.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T04:10:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:10:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:10:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:10:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:10:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:55.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:10:46' 2024-04-25T04:10:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:10:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:10:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:10:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:10:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:10:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:10:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:10:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:10:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 04:10:46' 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:10:46' 2024-04-25T04:10:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:10:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:10:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:10:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:10:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:10:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T04:10:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:10:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T04:10:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:10:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:10:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary4177df09f-9646-4591-868a-f910882375b6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:10:46' 2024-04-25T04:10:55.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:10:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:10:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:10:46' 2024-04-25T04:10:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:10:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:10:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:10:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:10:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T04:10:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:10:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T04:10:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:10:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:10:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:10:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:10:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T04:10:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:56.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:10:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T04:10:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:10:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:10:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T04:10:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T04:10:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T04:10:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:10:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T04:10:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:10:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:10:56.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T04:10:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:10:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:11:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=bda3e095eac904089de98cb16679a067 2024-04-25T04:11:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T04:11:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=94724bf3f56387884d469b3f184d9daa 2024-04-25T04:11:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T04:11:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 94724bf3f56387884d469b3f184d9daa != \9\4\7\2\4\b\f\3\f\5\6\3\8\7\8\8\4\d\4\6\9\b\3\f\1\8\4\d\9\d\a\a ]] 2024-04-25T04:11:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ bda3e095eac904089de98cb16679a067 != \b\d\a\3\e\0\9\5\e\a\c\9\0\4\0\8\9\d\e\9\8\c\b\1\6\6\7\9\a\0\6\7 ]] 2024-04-25T04:11:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=aa6cd30be68971db17cf70ef2e0ecb5f 2024-04-25T04:11:44.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T04:11:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ aa6cd30be68971db17cf70ef2e0ecb5f != \a\a\6\c\d\3\0\b\e\6\8\9\7\1\d\b\1\7\c\f\7\0\e\f\2\e\0\e\c\b\5\f ]] 2024-04-25T04:11:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=724f6ab49b347a1d6184eed73b347d7c 2024-04-25T04:11:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T04:11:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 724f6ab49b347a1d6184eed73b347d7c != \7\2\4\f\6\a\b\4\9\b\3\4\7\a\1\d\6\1\8\4\e\e\d\7\3\b\3\4\7\d\7\c ]] 2024-04-25T04:11:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=a862ed835a4d735555eaa860e874f3b6 2024-04-25T04:11:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T04:11:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ a862ed835a4d735555eaa860e874f3b6 != \a\8\6\2\e\d\8\3\5\a\4\d\7\3\5\5\5\5\e\a\a\8\6\0\e\8\7\4\f\3\b\6 ]] 2024-04-25T04:11:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=c01110ed2409da0e46966ec539c780f1 2024-04-25T04:11:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T04:11:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ c01110ed2409da0e46966ec539c780f1 != \c\0\1\1\1\0\e\d\2\4\0\9\d\a\0\e\4\6\9\6\6\e\c\5\3\9\c\7\8\0\f\1 ]] 2024-04-25T04:11:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=a976023555e0094dbf6337d02f65a1ba 2024-04-25T04:11:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T04:11:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ a976023555e0094dbf6337d02f65a1ba != \a\9\7\6\0\2\3\5\5\5\e\0\0\9\4\d\b\f\6\3\3\7\d\0\2\f\6\5\a\1\b\a ]] 2024-04-25T04:11:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109203 2024-04-25T04:11:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109205 2024-04-25T04:11:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109207 2024-04-25T04:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=df2cadf3b682e1cd6a30cec5ceaecd15 2024-04-25T04:11:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T04:11:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ df2cadf3b682e1cd6a30cec5ceaecd15 != \d\f\2\c\a\d\f\3\b\6\8\2\e\1\c\d\6\a\3\0\c\e\c\5\c\e\a\e\c\d\1\5 ]] 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109209 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109211 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109213 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109215 2024-04-25T04:11:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=e0852dd5662ea2879d7dc8d5a7ccfbde 2024-04-25T04:11:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T04:11:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ e0852dd5662ea2879d7dc8d5a7ccfbde != \e\0\8\5\2\d\d\5\6\6\2\e\a\2\8\7\9\d\7\d\c\8\d\5\a\7\c\c\f\b\d\e ]] 2024-04-25T04:11:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109218 2024-04-25T04:11:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:11:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 109219 2024-04-25T04:11:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=720334046a482935d8a93060cc0b65e5 2024-04-25T04:11:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T04:11:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 720334046a482935d8a93060cc0b65e5 != \7\2\0\3\3\4\0\4\6\a\4\8\2\9\3\5\d\8\a\9\3\0\6\0\c\c\0\b\6\5\e\5 ]] 2024-04-25T04:11:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:11:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:11:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:11:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:11:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T04:11:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:11:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:11:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:11:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:11:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:11:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:11:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:11:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 68825e16-9975-4692-b7f2-c9e1c17fdb0d 2024-04-25T04:11:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:11:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3699916.8,"bytes_per_snapshot":73998336.0,"last_snapshot_bytes":73998336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"} 2024-04-25T04:11:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:11:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:11:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:11:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:11:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:11:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3699916.8,"bytes_per_snapshot":73998336.0,"last_snapshot_bytes":73998336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"} 2024-04-25T04:11:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:11:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:11:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:11:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:11:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:11:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:11:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:11:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:11:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:11:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:11:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:11:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary168825e16-9975-4692-b7f2-c9e1c17fdb0dup+replayingreplaying, {"bytes_per_second":3699916.8,"bytes_per_snapshot":73998336.0,"last_snapshot_bytes":73998336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+replayingreplaying, {"bytes_per_second":3122585.6,"bytes_per_snapshot":62451712.0,"last_snapshot_bytes":62451712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+replayingreplaying, {"bytes_per_second":3170918.4,"bytes_per_snapshot":63418368.0,"last_snapshot_bytes":63418368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary4177df09f-9646-4591-868a-f910882375b6up+replayingreplaying, {"bytes_per_second":3250585.6,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+replayingreplaying, {"bytes_per_second":3698483.2,"bytes_per_snapshot":73969664.0,"last_snapshot_bytes":73969664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+replayingreplaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+replayingreplaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:11:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:11:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:11:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:11:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:11:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:11:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:11:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:11:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:11:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T04:11:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:11:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:11:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T04:11:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T04:11:53.190 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:11:53.184+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667bc61a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:11:53.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:11:53.184+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/68825e16-9975-4692-b7f2-c9e1c17fdb0d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:11:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:11:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T04:11:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:11:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:11:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:11:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T04:11:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T04:11:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:11:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:11:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T04:11:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:11:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:11:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:11:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:11:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:11:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: bae18c92-73f9-406c-a794-546ed67a522b 2024-04-25T04:11:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:11:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3122585.6,"bytes_per_snapshot":62451712.0,"last_snapshot_bytes":62451712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"} 2024-04-25T04:11:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:11:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:11:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:11:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:11:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:11:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3122585.6,"bytes_per_snapshot":62451712.0,"last_snapshot_bytes":62451712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"} 2024-04-25T04:11:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:11:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:11:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:11:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:11:55.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:11:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:11:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:11:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:11:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:11:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:11:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:11:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary2bae18c92-73f9-406c-a794-546ed67a522bup+replayingreplaying, {"bytes_per_second":3122585.6,"bytes_per_snapshot":62451712.0,"last_snapshot_bytes":62451712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+replayingreplaying, {"bytes_per_second":3170918.4,"bytes_per_snapshot":63418368.0,"last_snapshot_bytes":63418368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary4177df09f-9646-4591-868a-f910882375b6up+replayingreplaying, {"bytes_per_second":3250585.6,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+replayingreplaying, {"bytes_per_second":3698483.2,"bytes_per_snapshot":73969664.0,"last_snapshot_bytes":73969664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+replayingreplaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+replayingreplaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:11:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:11:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:11:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:11:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:11:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:11:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:11:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:11:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:11:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T04:11:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:11:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:11:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T04:11:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T04:11:56.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:11:56.172+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966447f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:11:56.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:11:56.172+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508c80 [3/bae18c92-73f9-406c-a794-546ed67a522b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:11:57.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:11:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T04:11:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:11:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:11:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:11:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T04:11:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T04:11:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:11:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:11:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:11:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:11:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:11:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:11:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d848cdb9-de5e-4504-9a7c-224a846e7c5e 2024-04-25T04:11:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:11:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3170918.4,"bytes_per_snapshot":63418368.0,"last_snapshot_bytes":63418368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"} 2024-04-25T04:11:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:11:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:11:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:11:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:11:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:11:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3170918.4,"bytes_per_snapshot":63418368.0,"last_snapshot_bytes":63418368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"} 2024-04-25T04:11:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:11:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:11:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:11:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:11:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:11:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:11:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:11:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:11:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:11:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:11:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary3d848cdb9-de5e-4504-9a7c-224a846e7c5eup+replayingreplaying, {"bytes_per_second":3170918.4,"bytes_per_snapshot":63418368.0,"last_snapshot_bytes":63418368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018250,"remote_snapshot_timestamp":1714018250,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary4177df09f-9646-4591-868a-f910882375b6up+replayingreplaying, {"bytes_per_second":3250585.6,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+replayingreplaying, {"bytes_per_second":3698483.2,"bytes_per_snapshot":73969664.0,"last_snapshot_bytes":73969664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+replayingreplaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+replayingreplaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:11:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:11:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:11:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:11:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:11:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:11:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:11:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:11:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:11:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:11:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T04:11:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:11:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:11:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:11:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T04:11:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T04:11:59.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:11:59.089+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668482340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:11:59.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:11:59.089+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/d848cdb9-de5e-4504-9a7c-224a846e7c5e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:00.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T04:12:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T04:12:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:12:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:12:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T04:12:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:12:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:12:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 177df09f-9646-4591-868a-f910882375b6 2024-04-25T04:12:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:12:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3250585.6,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"} 2024-04-25T04:12:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:12:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:12:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:12:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:12:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:12:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3250585.6,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"} 2024-04-25T04:12:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:12:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:12:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:12:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:12:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:12:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:12:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:12:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:12:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:12:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary4177df09f-9646-4591-868a-f910882375b6up+replayingreplaying, {"bytes_per_second":3250585.6,"bytes_per_snapshot":65011712.0,"last_snapshot_bytes":65011712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+replayingreplaying, {"bytes_per_second":3698483.2,"bytes_per_snapshot":73969664.0,"last_snapshot_bytes":73969664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+replayingreplaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+replayingreplaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:12:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:12:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:12:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:01.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:12:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:12:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:01.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:12:01.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T04:12:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:12:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:01.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T04:12:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T04:12:02.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:02.145+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664d111e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:12:02.150 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:02.145+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/177df09f-9646-4591-868a-f910882375b6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T04:12:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T04:12:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T04:12:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:12:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:03.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:12:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T04:12:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:12:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:12:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 00618b50-8806-4f74-87ae-e83a03ea5580 2024-04-25T04:12:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:12:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3698483.2,"bytes_per_snapshot":73969664.0,"last_snapshot_bytes":73969664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"} 2024-04-25T04:12:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:12:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:12:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:12:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:12:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:12:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3698483.2,"bytes_per_snapshot":73969664.0,"last_snapshot_bytes":73969664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"} 2024-04-25T04:12:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:12:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:12:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:12:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:12:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:12:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:12:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:12:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary500618b50-8806-4f74-87ae-e83a03ea5580up+replayingreplaying, {"bytes_per_second":3698483.2,"bytes_per_snapshot":73969664.0,"last_snapshot_bytes":73969664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018253,"remote_snapshot_timestamp":1714018253,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+replayingreplaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+replayingreplaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:12:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:12:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:12:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:12:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:12:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T04:12:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:12:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T04:12:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T04:12:04.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:04.894+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596677024e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:12:04.899 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:04.894+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/00618b50-8806-4f74-87ae-e83a03ea5580] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T04:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T04:12:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:12:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T04:12:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:12:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:12:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a70e20c2-c6c2-4ab7-83af-dacabb40573f 2024-04-25T04:12:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:12:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"} 2024-04-25T04:12:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:12:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:12:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:12:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:12:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:12:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"} 2024-04-25T04:12:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:12:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:12:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:12:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:12:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:12:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:12:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:07.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:12:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary6a70e20c2-c6c2-4ab7-83af-dacabb40573fup+replayingreplaying, {"bytes_per_second":3603456.0,"bytes_per_snapshot":72069120.0,"last_snapshot_bytes":72069120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018249,"remote_snapshot_timestamp":1714018249,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+replayingreplaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:12:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:12:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:12:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:12:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:12:07.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:12:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T04:12:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:12:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T04:12:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T04:12:07.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:07.903+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596678791e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:12:07.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:07.903+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508780 [3/a70e20c2-c6c2-4ab7-83af-dacabb40573f] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T04:12:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T04:12:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T04:12:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:12:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T04:12:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:12:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:12:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 626c9677-cb63-430c-925d-37f989447e3b 2024-04-25T04:12:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:12:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"} 2024-04-25T04:12:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:12:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:12:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:12:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:12:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:12:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"} 2024-04-25T04:12:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:12:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:12:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:12:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:12:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:12:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:12:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:12:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:12:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:12:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary7626c9677-cb63-430c-925d-37f989447e3bup+replayingreplaying, {"bytes_per_second":3233792.0,"bytes_per_snapshot":64675840.0,"last_snapshot_bytes":64675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018252,"remote_snapshot_timestamp":1714018252,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:12:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:12:10.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:12:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:12:10.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:12:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:12:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T04:12:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:12:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T04:12:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T04:12:10.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:10.886+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668483040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:12:10.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:10.886+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/626c9677-cb63-430c-925d-37f989447e3b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T04:12:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T04:12:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T04:12:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:12:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T04:12:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:12.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:12.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:12:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T04:12:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:12:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:12:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 50cb1558-010e-4201-9914-6ffa38efd173 2024-04-25T04:12:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:12:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"} 2024-04-25T04:12:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:12:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:12:13.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:12:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:12:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:12:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"} 2024-04-25T04:12:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:12:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:12:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:12:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:12:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:12:13.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:12:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:12:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:12:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:12:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary850cb1558-010e-4201-9914-6ffa38efd173up+replayingreplaying, {"bytes_per_second":3402752.0,"bytes_per_snapshot":68055040.0,"last_snapshot_bytes":68055040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:12:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:12:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:12:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:12:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:12:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:13.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:12:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T04:12:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:12:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T04:12:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T04:12:13.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:13.888+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596679c6ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:12:13.893 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:13.888+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/50cb1558-010e-4201-9914-6ffa38efd173] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T04:12:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:15.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T04:12:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:12:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T04:12:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:16.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:16.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:12:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:12:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d 2024-04-25T04:12:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:12:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"} 2024-04-25T04:12:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:12:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:11:46' 2024-04-25T04:12:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:12:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:12:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:12:16.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"} 2024-04-25T04:12:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:12:16.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:12:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:12:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:12:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:12:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 04:11:46' 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:11:46' 2024-04-25T04:12:16.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:12:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":3709747.2,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46image-primary96ecf8df4-d815-42e6-a1e9-01ac5e2ad44dup+replayingreplaying, {"bytes_per_second":3339468.8,"bytes_per_snapshot":66789376.0,"last_snapshot_bytes":66789376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018251,"remote_snapshot_timestamp":1714018251,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:11:46' 2024-04-25T04:12:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:12:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:12:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:11:46' 2024-04-25T04:12:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:12:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:12:16.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T04:12:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:12:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T04:12:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T04:12:16.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:16.908+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668603380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:12:16.913 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:16.908+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508a00 [3/6ecf8df4-d815-42e6-a1e9-01ac5e2ad44d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T04:12:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T04:12:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:12:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T04:12:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:12:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:12:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:12:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:12:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 5f07675c-6b5b-459a-aee0-c294855011a0 2024-04-25T04:12:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:12:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"} 2024-04-25T04:12:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:12:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:12:16' 2024-04-25T04:12:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:12:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:12:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:12:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"} 2024-04-25T04:12:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:12:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:12:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:12:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:12:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:12:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:12:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:12:16' 2024-04-25T04:12:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 04:12:16' 2024-04-25T04:12:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:12:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:12:16' 2024-04-25T04:12:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:12:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:12:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:12:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary105f07675c-6b5b-459a-aee0-c294855011a0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":74194944.0,"last_snapshot_bytes":74194944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018256,"remote_snapshot_timestamp":1714018256,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:12:16' 2024-04-25T04:12:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:12:19.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:12:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:12:16' 2024-04-25T04:12:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:12:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:12:19.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:12:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:12:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T04:12:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:12:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:19.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T04:12:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T04:12:19.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:19.897+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966810da00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:12:19.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:12:19.897+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/5f07675c-6b5b-459a-aee0-c294855011a0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:12:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:12:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T04:12:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:12:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:12:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T04:12:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T04:12:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T04:12:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T04:12:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T04:12:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T04:12:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T04:12:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T04:12:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T04:12:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T04:12:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T04:12:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T04:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T04:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:22.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2e2f3b208a87 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2e2f3b208a87 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:21 2024 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:21 2024 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:21 2024 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: d2ee79af-f305-4593-9ec9-1143bab1b9f3 2024-04-25T04:12:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T04:12:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T04:12:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T04:12:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 9189dbf1-3ffb-4948-a1b6-eb472256044b 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T04:12:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T04:12:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T04:12:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T04:12:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T04:12:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T04:12:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T04:12:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T04:12:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T04:12:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1984 2024-04-25T04:12:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T04:12:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T04:12:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2e3f1548776e 2024-04-25T04:12:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2e3f1548776e 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:23 2024 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:23 2024 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:23 2024 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2 2024-04-25T04:12:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T04:12:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T04:12:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T04:12:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: c337530e-7993-45d7-b617-6326b04a3154 2024-04-25T04:12:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:23.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:23.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:23.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T04:12:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T04:12:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:23.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T04:12:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T04:12:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T04:12:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:24.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T04:12:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1986 2024-04-25T04:12:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T04:12:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T04:12:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2e4e3b4eb6c1 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2e4e3b4eb6c1 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:23 2024 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:23 2024 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:23 2024 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:12:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T04:12:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T04:12:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T04:12:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 7c2acdb6-aa85-4d54-83a5-bb1227ec0969 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T04:12:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T04:12:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T04:12:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T04:12:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T04:12:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T04:12:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T04:12:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T04:12:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T04:12:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1988 2024-04-25T04:12:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T04:12:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T04:12:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2e5dd7aea319 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2e5dd7aea319 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:24 2024 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:24 2024 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:24 2024 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: a0673505-1191-4fe5-8857-9295de464ef9 2024-04-25T04:12:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T04:12:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T04:12:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:25.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T04:12:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 0f5be6ad-69e6-4619-b438-dbd5e2e19c44 2024-04-25T04:12:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T04:12:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T04:12:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T04:12:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T04:12:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T04:12:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T04:12:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1990 2024-04-25T04:12:26.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2e6cb5bb582d 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2e6cb5bb582d 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:25 2024 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:25 2024 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:25 2024 2024-04-25T04:12:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 8c02d0e5-43d5-4da4-97f6-3e281c65f08c 2024-04-25T04:12:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T04:12:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T04:12:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T04:12:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 7003e144-ba72-4fee-92e1-30bf7afac71a 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T04:12:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T04:12:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T04:12:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T04:12:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:27.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T04:12:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T04:12:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1993 2024-04-25T04:12:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1992 2024-04-25T04:12:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:27.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T04:12:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T04:12:27.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2e7e9f8a851f 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2e7e9f8a851f 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:27 2024 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:27 2024 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:27 2024 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 1939701d-5c04-4503-9ffe-886f9d016767 2024-04-25T04:12:27.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T04:12:27.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T04:12:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T04:12:27.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 071d2259-0164-48d7-94ae-ef7dc2948752 2024-04-25T04:12:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T04:12:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T04:12:28.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:28.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T04:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T04:12:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1995 2024-04-25T04:12:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T04:12:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1998 2024-04-25T04:12:29.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1996 2024-04-25T04:12:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T04:12:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T04:12:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2e8de5ec3efe 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2e8de5ec3efe 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:28 2024 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:28 2024 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:28 2024 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: e8f2676f-ec3c-4a70-bcbd-029ddf70a19b 2024-04-25T04:12:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T04:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T04:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T04:12:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 8ae9b2b7-6dca-4300-b2ce-66a77d012e1a 2024-04-25T04:12:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:29.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T04:12:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T04:12:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T04:12:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T04:12:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T04:12:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2000 2024-04-25T04:12:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 1999 2024-04-25T04:12:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:30.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:30.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T04:12:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:30.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T04:12:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2ea0d5f34b40 2024-04-25T04:12:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2ea0d5f34b40 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:30 2024 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:30 2024 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:30 2024 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 892fc3af-e16e-4e44-ac1e-1cd8b587f3c0 2024-04-25T04:12:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T04:12:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T04:12:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:30.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T04:12:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ebcadd4a-2a93-4b88-8613-cac6f6d0c1f4 2024-04-25T04:12:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:30.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T04:12:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T04:12:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T04:12:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T04:12:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T04:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T04:12:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T04:12:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2002 2024-04-25T04:12:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2003 2024-04-25T04:12:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2004 2024-04-25T04:12:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T04:12:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T04:12:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2eb54adecbbd 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2eb54adecbbd 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:31 2024 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:31 2024 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:31 2024 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:12:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T04:12:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T04:12:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T04:12:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 10074e9f-eb8f-4603-a0d0-387cfe831acf 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T04:12:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T04:12:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T04:12:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:12:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:12:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:12:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T04:12:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:32.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T04:12:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:12:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2006 2024-04-25T04:12:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2007 2024-04-25T04:12:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T04:12:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 2ec58bc2c462 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.2ec58bc2c462 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:12:32 2024 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:12:32 2024 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:12:32 2024 2024-04-25T04:12:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:12:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:12:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 64606122-008b-4df9-8a40-7df180c17084 2024-04-25T04:12:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:12:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:12:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:12:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T04:12:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T04:12:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:12:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T04:12:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:12:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:12:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:12:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: f501fb7b-ea31-4999-89b2-da925e34c638 2024-04-25T04:12:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:12:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:12:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:12:32.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:12:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:12:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:12:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:12:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T04:12:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T04:12:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:12:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:12:32.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T04:12:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:12:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 113763 2024-04-25T04:12:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T04:12:32.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T04:12:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:12:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T04:12:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2011 2024-04-25T04:12:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2009 2024-04-25T04:12:33.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2010 2024-04-25T04:12:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:12:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2013 2024-04-25T04:12:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:34.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2012 2024-04-25T04:12:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:12:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:12:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2016 2024-04-25T04:12:35.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2015 2024-04-25T04:12:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2014 2024-04-25T04:12:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:12:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:12:35.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:12:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:36.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:12:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:36.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:36.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:36.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2017 2024-04-25T04:12:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2018 2024-04-25T04:12:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2020 2024-04-25T04:12:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:12:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2021 2024-04-25T04:12:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2019 2024-04-25T04:12:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2022 2024-04-25T04:12:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:38.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:12:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:12:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2023 2024-04-25T04:12:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2026 2024-04-25T04:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2024 2024-04-25T04:12:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2025 2024-04-25T04:12:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:12:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:12:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:12:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:12:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2028 2024-04-25T04:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2027 2024-04-25T04:12:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:40.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2031 2024-04-25T04:12:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2029 2024-04-25T04:12:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2030 2024-04-25T04:12:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:41.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:12:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2032 2024-04-25T04:12:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2033 2024-04-25T04:12:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:12:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:12:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2036 2024-04-25T04:12:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2035 2024-04-25T04:12:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:12:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:12:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2038 2024-04-25T04:12:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:12:44.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:12:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:44.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2034 2024-04-25T04:12:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2040 2024-04-25T04:12:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2039 2024-04-25T04:12:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:12:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2037 2024-04-25T04:12:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2041 2024-04-25T04:12:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:12:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:12:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2042 2024-04-25T04:12:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:12:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:12:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2044 2024-04-25T04:12:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2043 2024-04-25T04:12:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2045 2024-04-25T04:12:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:12:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:12:48.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2046 2024-04-25T04:12:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2049 2024-04-25T04:12:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2047 2024-04-25T04:12:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:12:50.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2050 2024-04-25T04:12:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:12:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:12:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2048 2024-04-25T04:12:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2051 2024-04-25T04:12:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:12:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:12:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2052 2024-04-25T04:12:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:12:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:12:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2054 2024-04-25T04:12:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2053 2024-04-25T04:12:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2056 2024-04-25T04:12:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2055 2024-04-25T04:12:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:12:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:12:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:12:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2058 2024-04-25T04:12:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2057 2024-04-25T04:12:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:12:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:12:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:12:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:12:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2060 2024-04-25T04:12:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:12:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:12:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:12:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2059 2024-04-25T04:12:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2062 2024-04-25T04:12:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2061 2024-04-25T04:12:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:12:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:12:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:12:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:12:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:12:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:12:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2064 2024-04-25T04:12:57.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2063 2024-04-25T04:12:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2065 2024-04-25T04:12:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2066 2024-04-25T04:12:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:12:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:12:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:12:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:12:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:12:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:12:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:12:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:58.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:12:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:12:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:12:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:12:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:12:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2068 2024-04-25T04:12:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2067 2024-04-25T04:12:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:12:59.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:12:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:12:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:12:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:12:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:12:59.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2070 2024-04-25T04:13:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2069 2024-04-25T04:13:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2072 2024-04-25T04:13:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2071 2024-04-25T04:13:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2074 2024-04-25T04:13:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2073 2024-04-25T04:13:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2076 2024-04-25T04:13:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2075 2024-04-25T04:13:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:04.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2078 2024-04-25T04:13:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2077 2024-04-25T04:13:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2080 2024-04-25T04:13:05.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2079 2024-04-25T04:13:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2082 2024-04-25T04:13:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2081 2024-04-25T04:13:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2084 2024-04-25T04:13:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2083 2024-04-25T04:13:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2085 2024-04-25T04:13:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2086 2024-04-25T04:13:08.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:08.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:08.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2087 2024-04-25T04:13:09.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2088 2024-04-25T04:13:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2089 2024-04-25T04:13:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2090 2024-04-25T04:13:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:11.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2091 2024-04-25T04:13:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2092 2024-04-25T04:13:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2094 2024-04-25T04:13:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2093 2024-04-25T04:13:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2096 2024-04-25T04:13:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2095 2024-04-25T04:13:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:13.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:13.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:14.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:14.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2098 2024-04-25T04:13:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2097 2024-04-25T04:13:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2100 2024-04-25T04:13:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:16.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:16.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2099 2024-04-25T04:13:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2101 2024-04-25T04:13:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2102 2024-04-25T04:13:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2104 2024-04-25T04:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2103 2024-04-25T04:13:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2106 2024-04-25T04:13:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2105 2024-04-25T04:13:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2108 2024-04-25T04:13:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2107 2024-04-25T04:13:19.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:19.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2109 2024-04-25T04:13:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2110 2024-04-25T04:13:20.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2111 2024-04-25T04:13:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2112 2024-04-25T04:13:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:22.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2114 2024-04-25T04:13:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2113 2024-04-25T04:13:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2115 2024-04-25T04:13:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2116 2024-04-25T04:13:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:24.602 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:13:24.597+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667962400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:13:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2117 2024-04-25T04:13:25.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2118 2024-04-25T04:13:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2120 2024-04-25T04:13:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2119 2024-04-25T04:13:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2122 2024-04-25T04:13:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2121 2024-04-25T04:13:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:27.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2124 2024-04-25T04:13:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:27.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2123 2024-04-25T04:13:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2126 2024-04-25T04:13:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2125 2024-04-25T04:13:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2127 2024-04-25T04:13:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2128 2024-04-25T04:13:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:29.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:29.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:29.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2129 2024-04-25T04:13:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2130 2024-04-25T04:13:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2132 2024-04-25T04:13:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2131 2024-04-25T04:13:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2134 2024-04-25T04:13:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:33.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2133 2024-04-25T04:13:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:33.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:13:33.452+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664ea2800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:13:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2137 2024-04-25T04:13:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:33.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2136 2024-04-25T04:13:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2135 2024-04-25T04:13:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:34.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2138 2024-04-25T04:13:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:35.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2140 2024-04-25T04:13:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2139 2024-04-25T04:13:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2142 2024-04-25T04:13:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2141 2024-04-25T04:13:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2144 2024-04-25T04:13:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2145 2024-04-25T04:13:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:38.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2143 2024-04-25T04:13:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2146 2024-04-25T04:13:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2147 2024-04-25T04:13:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:38.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2148 2024-04-25T04:13:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:40.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2150 2024-04-25T04:13:40.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2149 2024-04-25T04:13:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:41.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:41.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:41.553 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:13:41.547+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667aba800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:13:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2153 2024-04-25T04:13:41.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2152 2024-04-25T04:13:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2151 2024-04-25T04:13:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2156 2024-04-25T04:13:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2155 2024-04-25T04:13:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2154 2024-04-25T04:13:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2157 2024-04-25T04:13:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:43.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2158 2024-04-25T04:13:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2160 2024-04-25T04:13:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2159 2024-04-25T04:13:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:45.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2161 2024-04-25T04:13:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2163 2024-04-25T04:13:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2164 2024-04-25T04:13:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2162 2024-04-25T04:13:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:46.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2165 2024-04-25T04:13:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2166 2024-04-25T04:13:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2167 2024-04-25T04:13:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:48.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2168 2024-04-25T04:13:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2169 2024-04-25T04:13:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2170 2024-04-25T04:13:50.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2173 2024-04-25T04:13:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2174 2024-04-25T04:13:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2172 2024-04-25T04:13:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2171 2024-04-25T04:13:52.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:13:52.460+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667599400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:13:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2176 2024-04-25T04:13:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2175 2024-04-25T04:13:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:53.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2177 2024-04-25T04:13:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:53.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2178 2024-04-25T04:13:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2179 2024-04-25T04:13:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2180 2024-04-25T04:13:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:13:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:13:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:13:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:13:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:13:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:13:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:13:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:13:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:13:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:13:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:13:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:13:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2181 2024-04-25T04:13:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:13:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:13:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:13:56.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2182 2024-04-25T04:13:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:13:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:13:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:13:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2186 2024-04-25T04:13:57.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2185 2024-04-25T04:13:57.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:13:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:13:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:13:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:13:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:13:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:13:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2183 2024-04-25T04:13:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2187 2024-04-25T04:13:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:13:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:13:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:13:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:13:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:13:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:13:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:13:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:13:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:13:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:13:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2188 2024-04-25T04:13:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2184 2024-04-25T04:14:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2189 2024-04-25T04:14:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2190 2024-04-25T04:14:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2192 2024-04-25T04:14:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2191 2024-04-25T04:14:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2193 2024-04-25T04:14:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2194 2024-04-25T04:14:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:02.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:02.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2195 2024-04-25T04:14:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2196 2024-04-25T04:14:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2198 2024-04-25T04:14:04.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:04.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2197 2024-04-25T04:14:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:05.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2202 2024-04-25T04:14:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2200 2024-04-25T04:14:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2199 2024-04-25T04:14:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2201 2024-04-25T04:14:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2203 2024-04-25T04:14:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:07.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2204 2024-04-25T04:14:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:08.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:08.413+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657df000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:08.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2205 2024-04-25T04:14:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2206 2024-04-25T04:14:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2209 2024-04-25T04:14:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2208 2024-04-25T04:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2207 2024-04-25T04:14:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2211 2024-04-25T04:14:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2210 2024-04-25T04:14:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2212 2024-04-25T04:14:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2213 2024-04-25T04:14:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:12.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2214 2024-04-25T04:14:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:12.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2216 2024-04-25T04:14:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2215 2024-04-25T04:14:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2217 2024-04-25T04:14:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2219 2024-04-25T04:14:14.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:14.408+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665774c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:14.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:14.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2218 2024-04-25T04:14:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:15.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:15.428+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665773c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2221 2024-04-25T04:14:15.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2220 2024-04-25T04:14:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2223 2024-04-25T04:14:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2222 2024-04-25T04:14:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:17.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2224 2024-04-25T04:14:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2225 2024-04-25T04:14:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:18.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2226 2024-04-25T04:14:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2227 2024-04-25T04:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:19.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:19.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:19.430+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966575ec00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2229 2024-04-25T04:14:19.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2228 2024-04-25T04:14:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2231 2024-04-25T04:14:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:20.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2230 2024-04-25T04:14:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2232 2024-04-25T04:14:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2233 2024-04-25T04:14:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:22.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:22.449+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657d2800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2234 2024-04-25T04:14:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:23.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2235 2024-04-25T04:14:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2236 2024-04-25T04:14:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2239 2024-04-25T04:14:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2238 2024-04-25T04:14:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:25.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2241 2024-04-25T04:14:25.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2240 2024-04-25T04:14:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2242 2024-04-25T04:14:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2243 2024-04-25T04:14:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2244 2024-04-25T04:14:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:27.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:28.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2237 2024-04-25T04:14:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2247 2024-04-25T04:14:28.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2246 2024-04-25T04:14:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2245 2024-04-25T04:14:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2248 2024-04-25T04:14:29.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2249 2024-04-25T04:14:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2250 2024-04-25T04:14:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:31.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:31.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:31.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2251 2024-04-25T04:14:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2252 2024-04-25T04:14:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2253 2024-04-25T04:14:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:31.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:32.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2256 2024-04-25T04:14:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2254 2024-04-25T04:14:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2255 2024-04-25T04:14:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2257 2024-04-25T04:14:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:34.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2258 2024-04-25T04:14:35.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:35.490+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596655e2400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:35.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:35.516+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966777a800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2263 2024-04-25T04:14:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:36.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2260 2024-04-25T04:14:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2259 2024-04-25T04:14:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2264 2024-04-25T04:14:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2261 2024-04-25T04:14:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2262 2024-04-25T04:14:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:38.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2265 2024-04-25T04:14:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:38.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:38.423+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665a6d800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:38.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:14:38.423+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657b3800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:14:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2266 2024-04-25T04:14:39.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2267 2024-04-25T04:14:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2269 2024-04-25T04:14:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2268 2024-04-25T04:14:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2271 2024-04-25T04:14:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2270 2024-04-25T04:14:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:40.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:41.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:41.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2272 2024-04-25T04:14:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2273 2024-04-25T04:14:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2274 2024-04-25T04:14:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:42.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:42.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:14:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:14:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:14:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:14:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:14:42.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:14:42.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2275 2024-04-25T04:14:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2276 2024-04-25T04:14:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2277 2024-04-25T04:14:43.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:14:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:14:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:14:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2278 2024-04-25T04:14:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:14:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:14:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:14:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2281 2024-04-25T04:14:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2280 2024-04-25T04:14:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2279 2024-04-25T04:14:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:45.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:14:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:14:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:14:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:46.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2282 2024-04-25T04:14:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2284 2024-04-25T04:14:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2283 2024-04-25T04:14:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2287 2024-04-25T04:14:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2286 2024-04-25T04:14:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:14:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:14:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:14:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2285 2024-04-25T04:14:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2288 2024-04-25T04:14:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 113897 2024-04-25T04:14:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114031 2024-04-25T04:14:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:14:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:14:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114165 2024-04-25T04:14:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114319 2024-04-25T04:14:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114474 2024-04-25T04:14:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:14:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:14:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:14:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:14:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:14:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:14:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:14:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:14:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:14:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114650 2024-04-25T04:14:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2290 2024-04-25T04:14:51.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2292 2024-04-25T04:14:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:53.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2289 2024-04-25T04:14:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2291 2024-04-25T04:14:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:14:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114825 2024-04-25T04:14:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114981 2024-04-25T04:14:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:14:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 115156 2024-04-25T04:14:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:14:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:14:57.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 113764 2024-04-25T04:17:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 113898 2024-04-25T04:17:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114032 2024-04-25T04:17:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:25.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114166 2024-04-25T04:17:26.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:26.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114320 2024-04-25T04:17:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114475 2024-04-25T04:17:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114651 2024-04-25T04:17:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114826 2024-04-25T04:17:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 114982 2024-04-25T04:17:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:32.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:17:32.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 115157 2024-04-25T04:17:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:17:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:17:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:17:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T04:17:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T04:17:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T04:17:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T04:17:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T04:17:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T04:17:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T04:17:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T04:17:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T04:17:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T04:17:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T04:17:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T04:17:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T04:17:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T04:17:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T04:17:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T04:17:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:17:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:17:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:17:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121131 2024-04-25T04:17:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T04:17:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T04:17:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T04:17:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T04:17:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T04:17:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T04:17:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T04:17:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T04:17:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T04:17:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:17:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T04:17:33.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:17:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:17:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:17:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:17:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:17:34.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:17:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:17:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:17:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:17:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:17:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:18:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=a668db97fa43d3ad4aa6399aea31fd0f 2024-04-25T04:18:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T04:18:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T04:18:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T04:18:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=047f41aa752d8ec6f174009edd68a447 2024-04-25T04:18:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T04:18:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=e4009025988065af00758ca28218bbc1 2024-04-25T04:18:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T04:18:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T04:18:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T04:18:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T04:18:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T04:18:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=a4066fa12255f3c49c42a9b78ecd2822 2024-04-25T04:18:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T04:18:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=eaddd5c20edc323f1db9da94c05b50bb 2024-04-25T04:18:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T04:18:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T04:18:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T04:18:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T04:18:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T04:18:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T04:18:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=102b96ed0067c021a255f28e597b9d2b 2024-04-25T04:18:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T04:18:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=d643ef72b08fddeba69919fee41e4829 2024-04-25T04:18:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T04:18:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T04:18:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T04:18:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T04:18:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T04:18:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=1bec1d215144df8e0f5bed4b3d1d0d69 2024-04-25T04:18:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T04:18:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=d4287241f9d1be5349f538b2c3a3daeb 2024-04-25T04:18:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T04:18:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T04:18:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T04:18:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T04:18:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:31.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T04:18:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:18:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:18:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=c4b14ff7629f3f77ce158aea519df711 2024-04-25T04:18:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T04:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:18:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:18:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:31.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T04:18:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T04:18:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:18:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 64606122-008b-4df9-8a40-7df180c17084 2024-04-25T04:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:32.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:33.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T04:18:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:33.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:18:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:33.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:18:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:18:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:18:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e8f2676f-ec3c-4a70-bcbd-029ddf70a19b 2024-04-25T04:18:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:18:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:18:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:18:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d2ee79af-f305-4593-9ec9-1143bab1b9f3 2024-04-25T04:18:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:33.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:18:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:18:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:18:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:18:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:33.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T04:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:18:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:18:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:18:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T04:18:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T04:18:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:18:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:18:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:18:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8c02d0e5-43d5-4da4-97f6-3e281c65f08c 2024-04-25T04:18:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:18:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 892fc3af-e16e-4e44-ac1e-1cd8b587f3c0 2024-04-25T04:18:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:18:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:34.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:18:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:18:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 64606122-008b-4df9-8a40-7df180c17084 2024-04-25T04:18:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:34.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:18:35.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:18:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:18:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:18:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:18:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:18:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:18:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a0673505-1191-4fe5-8857-9295de464ef9 2024-04-25T04:18:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:18:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2 2024-04-25T04:18:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:18:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1939701d-5c04-4503-9ffe-886f9d016767 2024-04-25T04:18:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:18:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:18:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e8f2676f-ec3c-4a70-bcbd-029ddf70a19b 2024-04-25T04:18:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:18:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d2ee79af-f305-4593-9ec9-1143bab1b9f3 2024-04-25T04:18:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:18:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:18:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:18:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8c02d0e5-43d5-4da4-97f6-3e281c65f08c 2024-04-25T04:18:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:18:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 892fc3af-e16e-4e44-ac1e-1cd8b587f3c0 2024-04-25T04:18:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:18:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:18:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:18:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:18:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a0673505-1191-4fe5-8857-9295de464ef9 2024-04-25T04:18:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:18:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1939701d-5c04-4503-9ffe-886f9d016767 2024-04-25T04:18:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2 2024-04-25T04:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:18:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:18:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:18:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:18:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:18:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:18:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:18:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 64606122-008b-4df9-8a40-7df180c17084 2024-04-25T04:18:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:18:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:18:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:18:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e8f2676f-ec3c-4a70-bcbd-029ddf70a19b 2024-04-25T04:18:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d2ee79af-f305-4593-9ec9-1143bab1b9f3 2024-04-25T04:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:18:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:40.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:18:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:18:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8c02d0e5-43d5-4da4-97f6-3e281c65f08c 2024-04-25T04:18:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:18:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 892fc3af-e16e-4e44-ac1e-1cd8b587f3c0 2024-04-25T04:18:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:18:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:18:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:18:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a0673505-1191-4fe5-8857-9295de464ef9 2024-04-25T04:18:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1939701d-5c04-4503-9ffe-886f9d016767 2024-04-25T04:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2 2024-04-25T04:18:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:18:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:18:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:18:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:18:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:16' 2024-04-25T04:18:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:18:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:46.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 64606122-008b-4df9-8a40-7df180c17084 2024-04-25T04:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:47.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:18:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:18:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:18:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:18:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e8f2676f-ec3c-4a70-bcbd-029ddf70a19b 2024-04-25T04:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:48.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:48.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:18:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d2ee79af-f305-4593-9ec9-1143bab1b9f3 2024-04-25T04:18:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:48.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:18:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:18:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:18:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:18:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T04:18:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T04:18:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T04:18:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:18:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 64606122-008b-4df9-8a40-7df180c17084 2024-04-25T04:18:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:18:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:18:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:48.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T04:18:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:18:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T04:18:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T04:18:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T04:18:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:18:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:18:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:18:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:18:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8c02d0e5-43d5-4da4-97f6-3e281c65f08c 2024-04-25T04:18:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 892fc3af-e16e-4e44-ac1e-1cd8b587f3c0 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:18:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:49.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:18:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:18:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:18:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T04:18:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T04:18:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:49.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T04:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e8f2676f-ec3c-4a70-bcbd-029ddf70a19b 2024-04-25T04:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:18:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d2ee79af-f305-4593-9ec9-1143bab1b9f3 2024-04-25T04:18:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:18:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:18:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T04:18:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:18:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:18:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T04:18:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T04:18:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T04:18:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T04:18:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:18:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T04:18:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:18:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:18:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:18:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:18:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:18:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:18:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:18:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2 2024-04-25T04:18:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:18:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1939701d-5c04-4503-9ffe-886f9d016767 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a0673505-1191-4fe5-8857-9295de464ef9 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:50.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T04:18:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T04:18:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:18:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:18:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:50.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:18:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:18:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:18:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:18:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:18:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 892fc3af-e16e-4e44-ac1e-1cd8b587f3c0 2024-04-25T04:18:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:18:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8c02d0e5-43d5-4da4-97f6-3e281c65f08c 2024-04-25T04:18:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:18:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:18:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:18:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:18:50.473 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T04:18:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:18:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T04:18:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T04:18:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T04:18:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T04:18:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T04:18:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:18:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T04:18:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:18:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:51.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T04:18:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:18:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2 2024-04-25T04:18:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:18:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a0673505-1191-4fe5-8857-9295de464ef9 2024-04-25T04:18:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1939701d-5c04-4503-9ffe-886f9d016767 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:51.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:18:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:18:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T04:18:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:18:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T04:18:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:18:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:18:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:18:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:18:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T04:18:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T04:18:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:52.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T04:18:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T04:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T04:18:53.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T04:18:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:18:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T04:18:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:18:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:53.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T04:18:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:18:53.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:18:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:18:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:18:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:18:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:18:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:18:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:18:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:18:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:18:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:54.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:18:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T04:18:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T04:18:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:18:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:18:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T04:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T04:18:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T04:18:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:18:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:18:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T04:18:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:18:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:18:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:18:46' 2024-04-25T04:18:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:18:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:18:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:18:55.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:18:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:18:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:18:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:18:55.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:18:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 04:18:46' 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:18:46' 2024-04-25T04:18:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:18:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:18:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:18:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:18:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:18:46' 2024-04-25T04:18:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:18:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:18:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:18:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:18:46' 2024-04-25T04:18:56.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:18:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:18:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T04:18:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:18:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T04:18:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:18:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:18:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:18:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T04:18:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:18:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T04:18:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:18:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:18:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T04:18:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T04:18:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:18:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:18:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T04:18:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:18:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:19:42.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=a668db97fa43d3ad4aa6399aea31fd0f 2024-04-25T04:19:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T04:19:43.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ a668db97fa43d3ad4aa6399aea31fd0f != \a\6\6\8\d\b\9\7\f\a\4\3\d\3\a\d\4\a\a\6\3\9\9\a\e\a\3\1\f\d\0\f ]] 2024-04-25T04:19:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=d643ef72b08fddeba69919fee41e4829 2024-04-25T04:19:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T04:19:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ d643ef72b08fddeba69919fee41e4829 != \d\6\4\3\e\f\7\2\b\0\8\f\d\d\e\b\a\6\9\9\1\9\f\e\e\4\1\e\4\8\2\9 ]] 2024-04-25T04:19:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=1bec1d215144df8e0f5bed4b3d1d0d69 2024-04-25T04:19:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T04:19:45.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 1bec1d215144df8e0f5bed4b3d1d0d69 != \1\b\e\c\1\d\2\1\5\1\4\4\d\f\8\e\0\f\5\b\e\d\4\b\3\d\1\d\0\d\6\9 ]] 2024-04-25T04:19:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121132 2024-04-25T04:19:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=d4287241f9d1be5349f538b2c3a3daeb 2024-04-25T04:19:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T04:19:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=a4066fa12255f3c49c42a9b78ecd2822 2024-04-25T04:19:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T04:19:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=c4b14ff7629f3f77ce158aea519df711 2024-04-25T04:19:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T04:19:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ d4287241f9d1be5349f538b2c3a3daeb != \d\4\2\8\7\2\4\1\f\9\d\1\b\e\5\3\4\9\f\5\3\8\b\2\c\3\a\3\d\a\e\b ]] 2024-04-25T04:19:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121134 2024-04-25T04:19:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ a4066fa12255f3c49c42a9b78ecd2822 != \a\4\0\6\6\f\a\1\2\2\5\5\f\3\c\4\9\c\4\2\a\9\b\7\8\e\c\d\2\8\2\2 ]] 2024-04-25T04:19:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ c4b14ff7629f3f77ce158aea519df711 != \c\4\b\1\4\f\f\7\6\2\9\f\3\f\7\7\c\e\1\5\8\a\e\a\5\1\9\d\f\7\1\1 ]] 2024-04-25T04:19:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=102b96ed0067c021a255f28e597b9d2b 2024-04-25T04:19:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T04:19:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 102b96ed0067c021a255f28e597b9d2b != \1\0\2\b\9\6\e\d\0\0\6\7\c\0\2\1\a\2\5\5\f\2\8\e\5\9\7\b\9\d\2\b ]] 2024-04-25T04:19:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=eaddd5c20edc323f1db9da94c05b50bb 2024-04-25T04:19:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T04:19:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ eaddd5c20edc323f1db9da94c05b50bb != \e\a\d\d\d\5\c\2\0\e\d\c\3\2\3\f\1\d\b\9\d\a\9\4\c\0\5\b\5\0\b\b ]] 2024-04-25T04:19:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=047f41aa752d8ec6f174009edd68a447 2024-04-25T04:19:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T04:19:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 047f41aa752d8ec6f174009edd68a447 != \0\4\7\f\4\1\a\a\7\5\2\d\8\e\c\6\f\1\7\4\0\0\9\e\d\d\6\8\a\4\4\7 ]] 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121136 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121138 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121140 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121142 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121144 2024-04-25T04:19:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121147 2024-04-25T04:19:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=e4009025988065af00758ca28218bbc1 2024-04-25T04:19:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T04:19:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ e4009025988065af00758ca28218bbc1 != \e\4\0\0\9\0\2\5\9\8\8\0\6\5\a\f\0\0\7\5\8\c\a\2\8\2\1\8\b\b\c\1 ]] 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 121148 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:19:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:19:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T04:19:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:19:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:19:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:19:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:19:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:19:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:19:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:19:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: d2ee79af-f305-4593-9ec9-1143bab1b9f3 2024-04-25T04:19:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:19:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3677184.0,"bytes_per_snapshot":73543680.0,"last_snapshot_bytes":73543680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"} 2024-04-25T04:19:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:19:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:19:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:19:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:19:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:19:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3677184.0,"bytes_per_snapshot":73543680.0,"last_snapshot_bytes":73543680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"} 2024-04-25T04:19:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:19:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:19:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:19:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:19:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:19:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:19:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:19:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:19:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:19:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:19:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:19:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:19:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:19:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:19:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:19:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1d2ee79af-f305-4593-9ec9-1143bab1b9f3up+replayingreplaying, {"bytes_per_second":3677184.0,"bytes_per_snapshot":73543680.0,"last_snapshot_bytes":73543680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+replayingreplaying, {"bytes_per_second":3673907.2,"bytes_per_snapshot":73478144.0,"last_snapshot_bytes":73478144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+replayingreplaying, {"bytes_per_second":3248742.4,"bytes_per_snapshot":64974848.0,"last_snapshot_bytes":64974848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+replayingreplaying, {"bytes_per_second":3685580.8,"bytes_per_snapshot":73711616.0,"last_snapshot_bytes":73711616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+replayingreplaying, {"bytes_per_second":3682099.2,"bytes_per_snapshot":73641984.0,"last_snapshot_bytes":73641984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+replayingreplaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+replayingreplaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:19:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:19:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:19:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:19:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:19:52.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:19:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:19:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:19:52.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:19:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T04:19:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:19:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:19:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T04:19:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T04:19:52.860 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:19:52.854+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667703040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:19:52.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:19:52.854+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508780 [3/d2ee79af-f305-4593-9ec9-1143bab1b9f3] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:19:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:19:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T04:19:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:19:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:19:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:19:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T04:19:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T04:19:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:19:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:19:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T04:19:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:19:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:19:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:19:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:19:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:19:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:19:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2 2024-04-25T04:19:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:19:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3673907.2,"bytes_per_snapshot":73478144.0,"last_snapshot_bytes":73478144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"} 2024-04-25T04:19:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:19:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:19:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:19:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:19:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:19:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3673907.2,"bytes_per_snapshot":73478144.0,"last_snapshot_bytes":73478144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"} 2024-04-25T04:19:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:19:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:19:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:19:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:19:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:19:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:19:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:19:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:19:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:19:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:19:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:19:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:19:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:19:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:19:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:19:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary27d64b8f9-8a4c-4a13-964d-6e3e3de6abd2up+replayingreplaying, {"bytes_per_second":3673907.2,"bytes_per_snapshot":73478144.0,"last_snapshot_bytes":73478144,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+replayingreplaying, {"bytes_per_second":3248742.4,"bytes_per_snapshot":64974848.0,"last_snapshot_bytes":64974848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+replayingreplaying, {"bytes_per_second":3685580.8,"bytes_per_snapshot":73711616.0,"last_snapshot_bytes":73711616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+replayingreplaying, {"bytes_per_second":3682099.2,"bytes_per_snapshot":73641984.0,"last_snapshot_bytes":73641984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+replayingreplaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+replayingreplaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:19:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:19:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:19:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:19:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:19:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:19:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:19:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:19:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:19:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T04:19:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:19:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:19:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T04:19:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T04:19:55.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:19:55.475+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667703380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:19:55.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:19:55.475+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/7d64b8f9-8a4c-4a13-964d-6e3e3de6abd2] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:19:56.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:19:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T04:19:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:19:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:19:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:19:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T04:19:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T04:19:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:19:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:19:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T04:19:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:19:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:19:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:19:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T04:19:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:19:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:19:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:19:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:19:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:19:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:19:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:19:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8cd5e2a5-80af-47c4-8ee2-3639e73aceed 2024-04-25T04:19:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:19:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3248742.4,"bytes_per_snapshot":64974848.0,"last_snapshot_bytes":64974848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"} 2024-04-25T04:19:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:19:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:19:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:19:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:19:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:19:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3248742.4,"bytes_per_snapshot":64974848.0,"last_snapshot_bytes":64974848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"} 2024-04-25T04:19:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:19:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:19:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:19:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:19:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:19:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:19:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:19:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:19:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:19:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:19:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary38cd5e2a5-80af-47c4-8ee2-3639e73aceedup+replayingreplaying, {"bytes_per_second":3248742.4,"bytes_per_snapshot":64974848.0,"last_snapshot_bytes":64974848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+replayingreplaying, {"bytes_per_second":3685580.8,"bytes_per_snapshot":73711616.0,"last_snapshot_bytes":73711616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+replayingreplaying, {"bytes_per_second":3682099.2,"bytes_per_snapshot":73641984.0,"last_snapshot_bytes":73641984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+replayingreplaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+replayingreplaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:19:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:19:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:19:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:19:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:19:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:19:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:19:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:19:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:19:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:19:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T04:19:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:19:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:19:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:19:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T04:19:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T04:19:58.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:19:58.481+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966447c9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:19:58.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:19:58.481+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508a00 [3/8cd5e2a5-80af-47c4-8ee2-3639e73aceed] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T04:20:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T04:20:00.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T04:20:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:00.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:20:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:20:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a0673505-1191-4fe5-8857-9295de464ef9 2024-04-25T04:20:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:20:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3685580.8,"bytes_per_snapshot":73711616.0,"last_snapshot_bytes":73711616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"} 2024-04-25T04:20:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:20:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:20:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:20:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:20:01.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:20:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3685580.8,"bytes_per_snapshot":73711616.0,"last_snapshot_bytes":73711616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"} 2024-04-25T04:20:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:20:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:20:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:20:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:20:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:20:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:01.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:20:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary4a0673505-1191-4fe5-8857-9295de464ef9up+replayingreplaying, {"bytes_per_second":3685580.8,"bytes_per_snapshot":73711616.0,"last_snapshot_bytes":73711616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+replayingreplaying, {"bytes_per_second":3682099.2,"bytes_per_snapshot":73641984.0,"last_snapshot_bytes":73641984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+replayingreplaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+replayingreplaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:20:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:20:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:20:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:20:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:20:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T04:20:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:20:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T04:20:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T04:20:01.690 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:01.684+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667a429c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:20:01.691 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:01.684+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/a0673505-1191-4fe5-8857-9295de464ef9] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T04:20:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T04:20:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:20:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:20:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8c02d0e5-43d5-4da4-97f6-3e281c65f08c 2024-04-25T04:20:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:20:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3682099.2,"bytes_per_snapshot":73641984.0,"last_snapshot_bytes":73641984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"} 2024-04-25T04:20:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:20:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:20:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:20:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:20:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:20:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3682099.2,"bytes_per_snapshot":73641984.0,"last_snapshot_bytes":73641984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"} 2024-04-25T04:20:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:20:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:20:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:20:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:20:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:20:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:20:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:20:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary58c02d0e5-43d5-4da4-97f6-3e281c65f08cup+replayingreplaying, {"bytes_per_second":3682099.2,"bytes_per_snapshot":73641984.0,"last_snapshot_bytes":73641984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+replayingreplaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+replayingreplaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:20:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:20:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:20:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:20:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:20:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:20:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T04:20:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:20:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T04:20:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T04:20:04.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:04.037+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c631e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:20:04.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:04.037+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/8c02d0e5-43d5-4da4-97f6-3e281c65f08c] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T04:20:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T04:20:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:06.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:20:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T04:20:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:20:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:20:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1939701d-5c04-4503-9ffe-886f9d016767 2024-04-25T04:20:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:20:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"} 2024-04-25T04:20:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:20:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:20:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:20:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:20:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:20:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"} 2024-04-25T04:20:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:20:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:20:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:20:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:20:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:20:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:20:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary61939701d-5c04-4503-9ffe-886f9d016767up+replayingreplaying, {"bytes_per_second":3679641.6,"bytes_per_snapshot":73592832.0,"last_snapshot_bytes":73592832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018732,"remote_snapshot_timestamp":1714018732,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+replayingreplaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:20:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:20:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:20:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:20:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:20:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:20:07.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T04:20:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:20:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T04:20:07.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T04:20:07.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:07.523+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966810c340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:20:07.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:07.523+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/1939701d-5c04-4503-9ffe-886f9d016767] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T04:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T04:20:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T04:20:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:20:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T04:20:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:09.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:09.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:20:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T04:20:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:20:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:20:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: e8f2676f-ec3c-4a70-bcbd-029ddf70a19b 2024-04-25T04:20:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:20:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"} 2024-04-25T04:20:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:20:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:20:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:20:10.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:20:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:20:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"} 2024-04-25T04:20:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:20:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:20:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:20:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:20:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:20:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:20:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:20:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:20:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:20:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary7e8f2676f-ec3c-4a70-bcbd-029ddf70a19bup+replayingreplaying, {"bytes_per_second":3226828.8,"bytes_per_snapshot":64536576.0,"last_snapshot_bytes":64536576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018729,"remote_snapshot_timestamp":1714018729,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:20:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:20:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:20:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:20:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:20:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T04:20:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:20:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T04:20:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T04:20:11.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:11.020+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c62000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:20:11.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:11.020+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/e8f2676f-ec3c-4a70-bcbd-029ddf70a19b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T04:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T04:20:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T04:20:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:20:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:20:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:20:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:20:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 892fc3af-e16e-4e44-ac1e-1cd8b587f3c0 2024-04-25T04:20:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:20:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"} 2024-04-25T04:20:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:20:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:20:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:20:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:20:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:20:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"} 2024-04-25T04:20:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:20:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:20:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:20:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:20:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:20:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:20:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:20:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":3509862.4,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary8892fc3af-e16e-4e44-ac1e-1cd8b587f3c0up+replayingreplaying, {"bytes_per_second":3321446.4,"bytes_per_snapshot":66428928.0,"last_snapshot_bytes":66428928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018730,"remote_snapshot_timestamp":1714018730,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:19:46' 2024-04-25T04:20:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:20:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:20:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:20:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:20:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:20:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T04:20:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:20:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T04:20:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T04:20:14.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:14.037+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664d11380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:20:14.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:14.037+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/892fc3af-e16e-4e44-ac1e-1cd8b587f3c0] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T04:20:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:15.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T04:20:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T04:20:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:20:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:16.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:20:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:20:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 10d6b9c5-5f64-41bc-a155-4751ecf73075 2024-04-25T04:20:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:20:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"} 2024-04-25T04:20:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:20:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:19:46' 2024-04-25T04:20:16.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:20:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:20:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:20:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3321856.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"} 2024-04-25T04:20:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:20:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:20:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:20:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:20:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:20:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:20:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:19:46' 2024-04-25T04:20:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 04:19:46' 2024-04-25T04:20:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:19:46' 2024-04-25T04:20:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:20:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:20:16image-primary910d6b9c5-5f64-41bc-a155-4751ecf73075up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66437120.0,"last_snapshot_bytes":66437120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018736,"remote_snapshot_timestamp":1714018736,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:20:16' 2024-04-25T04:20:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:20:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:20:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:20:16' 2024-04-25T04:20:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:20:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:20:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:20:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T04:20:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:20:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T04:20:16.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T04:20:17.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:17.050+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966810d380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:20:17.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:17.050+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/10d6b9c5-5f64-41bc-a155-4751ecf73075] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T04:20:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T04:20:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T04:20:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:20:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:20:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T04:20:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:20:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:20:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 64606122-008b-4df9-8a40-7df180c17084 2024-04-25T04:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"} 2024-04-25T04:20:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:20:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:20:16' 2024-04-25T04:20:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:20:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:20:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:20:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"} 2024-04-25T04:20:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:20:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:20:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:20:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:20:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:20:16' 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 04:20:16' 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:20:16' 2024-04-25T04:20:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:20:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:20:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:20:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1064606122-008b-4df9-8a40-7df180c17084up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714018728,"remote_snapshot_timestamp":1714018728,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:20:16' 2024-04-25T04:20:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:20:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:20:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:20:16' 2024-04-25T04:20:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:20:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:20:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:20:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:20:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:20:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T04:20:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:20:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T04:20:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T04:20:20.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:20.039+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667a4e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:20:20.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:20:20.039+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508c80 [3/64606122-008b-4df9-8a40-7df180c17084] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:20:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:20:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T04:20:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:20:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:20:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T04:20:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:21.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T04:20:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:21.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T04:20:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:21.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T04:20:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 330ad50ccc97 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.330ad50ccc97 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:21 2024 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:21 2024 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:21 2024 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ce19b877-ee6f-4303-8695-7d447d2f80f1 2024-04-25T04:20:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T04:20:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T04:20:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:22.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T04:20:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 817a295c-a400-4e2d-912a-a0ffbc5511d0 2024-04-25T04:20:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T04:20:22.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T04:20:22.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T04:20:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T04:20:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T04:20:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T04:20:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T04:20:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T04:20:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2314 2024-04-25T04:20:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T04:20:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 331a24d53e5a 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.331a24d53e5a 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:23 2024 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:23 2024 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:23 2024 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:20:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T04:20:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T04:20:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:23.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T04:20:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: f46a63c2-eed9-4484-bd1e-41a3b03acfc9 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T04:20:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T04:20:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T04:20:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T04:20:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T04:20:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T04:20:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T04:20:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T04:20:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2316 2024-04-25T04:20:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T04:20:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3329d0c9eacf 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3329d0c9eacf 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:23 2024 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:23 2024 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:23 2024 2024-04-25T04:20:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 1d7db7b9-4ab5-49e3-a40a-19ece53720ab 2024-04-25T04:20:24.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T04:20:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T04:20:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T04:20:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 1cd5e625-0f65-4897-a69f-7f6ab1d1486a 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T04:20:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T04:20:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T04:20:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T04:20:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T04:20:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2318 2024-04-25T04:20:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T04:20:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3338882505b6 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3338882505b6 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:24 2024 2024-04-25T04:20:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:24 2024 2024-04-25T04:20:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:24 2024 2024-04-25T04:20:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 33c83061-ffe0-4fdc-83ce-fadcd2db3404 2024-04-25T04:20:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T04:20:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T04:20:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T04:20:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: cdf41550-33e4-411c-9c12-8e1c1d09ac40 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:25.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T04:20:25.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T04:20:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T04:20:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:20:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T04:20:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T04:20:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T04:20:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T04:20:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T04:20:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T04:20:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2320 2024-04-25T04:20:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T04:20:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T04:20:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 33474433f849 2024-04-25T04:20:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.33474433f849 2024-04-25T04:20:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:26 2024 2024-04-25T04:20:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:26 2024 2024-04-25T04:20:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:26 2024 2024-04-25T04:20:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: b04734ef-e82a-406f-aabe-120b4b0d23e6 2024-04-25T04:20:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T04:20:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T04:20:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:26.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T04:20:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 70b08d24-a58b-4f6d-b160-eb9824971a56 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T04:20:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T04:20:27.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T04:20:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T04:20:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:27.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:27.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T04:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T04:20:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T04:20:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2322 2024-04-25T04:20:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T04:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2325 2024-04-25T04:20:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2323 2024-04-25T04:20:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:28.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T04:20:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T04:20:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 335834f01464 2024-04-25T04:20:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.335834f01464 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:27 2024 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:27 2024 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:27 2024 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 407d8be9-f19a-4331-91de-8e896ef24293 2024-04-25T04:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T04:20:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T04:20:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T04:20:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 390797d3-bc40-41ef-a3ff-5e375f099dba 2024-04-25T04:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T04:20:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T04:20:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T04:20:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T04:20:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T04:20:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T04:20:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2327 2024-04-25T04:20:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2326 2024-04-25T04:20:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:29.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T04:20:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T04:20:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 336cbe4bc0ae 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.336cbe4bc0ae 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:29 2024 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:29 2024 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:29 2024 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: da170bd5-4439-490d-9f61-ca4a7eeacd5f 2024-04-25T04:20:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T04:20:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T04:20:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T04:20:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 9265cb99-9c8f-4ea6-84b5-2bfb3a3210bc 2024-04-25T04:20:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T04:20:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T04:20:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T04:20:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T04:20:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:20:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T04:20:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T04:20:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T04:20:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T04:20:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2329 2024-04-25T04:20:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2330 2024-04-25T04:20:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T04:20:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 337d1192a82c 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.337d1192a82c 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:30 2024 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:30 2024 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:30 2024 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 937679fc-49d1-4466-96ec-80fcc44b3c02 2024-04-25T04:20:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:30.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T04:20:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T04:20:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T04:20:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: a08e2bf4-cba0-43f1-8e2e-2a09940f8b2d 2024-04-25T04:20:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T04:20:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T04:20:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T04:20:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:20:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T04:20:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T04:20:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2332 2024-04-25T04:20:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2333 2024-04-25T04:20:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T04:20:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 338e65244594 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.338e65244594 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:31 2024 2024-04-25T04:20:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:31 2024 2024-04-25T04:20:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:31 2024 2024-04-25T04:20:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 418dbbe0-c64c-4fab-89e3-76cc0e609f86 2024-04-25T04:20:31.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T04:20:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T04:20:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T04:20:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 4bd0907d-091a-48ad-8e33-2290fbb83a25 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:31.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T04:20:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T04:20:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:20:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:20:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:20:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:20:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T04:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T04:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:20:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T04:20:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T04:20:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2337 2024-04-25T04:20:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:32.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2335 2024-04-25T04:20:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:20:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2336 2024-04-25T04:20:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T04:20:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T04:20:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:20:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:20:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:20:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 33a3fb3d22b6 2024-04-25T04:20:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.33a3fb3d22b6 2024-04-25T04:20:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:20:32 2024 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:20:32 2024 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:20:32 2024 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: feaa7574-7ded-461d-b6e2-1af898d5e863 2024-04-25T04:20:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:20:32.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:20:32.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:20:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T04:20:32.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T04:20:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:20:32.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T04:20:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:20:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:20:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:20:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: a011c5c9-7b09-4e17-bf20-32eaecd8b473 2024-04-25T04:20:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:20:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:20:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:20:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:20:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:20:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:20:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:20:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T04:20:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T04:20:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:20:33.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T04:20:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:20:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 125611 2024-04-25T04:20:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T04:20:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:20:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T04:20:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T04:20:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:20:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T04:20:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2340 2024-04-25T04:20:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2341 2024-04-25T04:20:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2339 2024-04-25T04:20:33.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:20:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:20:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2342 2024-04-25T04:20:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2343 2024-04-25T04:20:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:20:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:20:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2345 2024-04-25T04:20:35.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2344 2024-04-25T04:20:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:20:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:20:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2347 2024-04-25T04:20:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2348 2024-04-25T04:20:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2346 2024-04-25T04:20:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:20:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:37.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2351 2024-04-25T04:20:37.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2350 2024-04-25T04:20:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2349 2024-04-25T04:20:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:20:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:20:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2353 2024-04-25T04:20:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2352 2024-04-25T04:20:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:39.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2355 2024-04-25T04:20:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2354 2024-04-25T04:20:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:20:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:20:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2357 2024-04-25T04:20:40.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2358 2024-04-25T04:20:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:40.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2356 2024-04-25T04:20:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:20:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:40.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:20:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2359 2024-04-25T04:20:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2360 2024-04-25T04:20:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2361 2024-04-25T04:20:41.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:20:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:20:42.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:20:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2362 2024-04-25T04:20:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2363 2024-04-25T04:20:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:42.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:43.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2364 2024-04-25T04:20:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:20:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:20:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2365 2024-04-25T04:20:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2366 2024-04-25T04:20:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2367 2024-04-25T04:20:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:20:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:20:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2368 2024-04-25T04:20:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2370 2024-04-25T04:20:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:45.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2369 2024-04-25T04:20:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:20:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2371 2024-04-25T04:20:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:20:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:20:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2373 2024-04-25T04:20:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:20:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:20:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2374 2024-04-25T04:20:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2372 2024-04-25T04:20:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:20:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:20:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2377 2024-04-25T04:20:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2375 2024-04-25T04:20:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2376 2024-04-25T04:20:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2379 2024-04-25T04:20:49.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:20:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2378 2024-04-25T04:20:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2380 2024-04-25T04:20:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:20:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2381 2024-04-25T04:20:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2382 2024-04-25T04:20:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:20:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:20:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2384 2024-04-25T04:20:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:20:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:20:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2383 2024-04-25T04:20:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2385 2024-04-25T04:20:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:53.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2387 2024-04-25T04:20:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:20:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2388 2024-04-25T04:20:54.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2386 2024-04-25T04:20:54.594 INFO:tasks.ceph.cluster1.osd.0.smithi173.stderr:2024-04-25T04:20:54.588+0000 7f4e05000700 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 93d after trimming was started 2024-04-25T04:20:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:55.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:20:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:20:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:20:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2390 2024-04-25T04:20:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2389 2024-04-25T04:20:55.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:20:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:20:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:20:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:20:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:20:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:20:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2392 2024-04-25T04:20:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2391 2024-04-25T04:20:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:20:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:20:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:20:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:20:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:20:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:20:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:20:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:20:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:20:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2394 2024-04-25T04:20:57.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2393 2024-04-25T04:20:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2395 2024-04-25T04:20:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:20:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:20:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:20:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:20:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:20:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:20:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:20:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:20:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:59.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:20:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:20:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:20:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:20:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:20:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:20:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:20:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2396 2024-04-25T04:21:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2398 2024-04-25T04:21:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2397 2024-04-25T04:21:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:00.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2400 2024-04-25T04:21:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2399 2024-04-25T04:21:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2401 2024-04-25T04:21:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2402 2024-04-25T04:21:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2404 2024-04-25T04:21:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2403 2024-04-25T04:21:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:03.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2405 2024-04-25T04:21:04.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2407 2024-04-25T04:21:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2406 2024-04-25T04:21:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2408 2024-04-25T04:21:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:06.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2410 2024-04-25T04:21:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2409 2024-04-25T04:21:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2414 2024-04-25T04:21:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:07.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2411 2024-04-25T04:21:07.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2412 2024-04-25T04:21:07.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2415 2024-04-25T04:21:07.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2413 2024-04-25T04:21:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2418 2024-04-25T04:21:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2416 2024-04-25T04:21:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:09.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2417 2024-04-25T04:21:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2419 2024-04-25T04:21:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2420 2024-04-25T04:21:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:11.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2422 2024-04-25T04:21:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2423 2024-04-25T04:21:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:11.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2421 2024-04-25T04:21:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2424 2024-04-25T04:21:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2425 2024-04-25T04:21:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:12.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:13.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2426 2024-04-25T04:21:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2427 2024-04-25T04:21:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2428 2024-04-25T04:21:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2430 2024-04-25T04:21:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2429 2024-04-25T04:21:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2435 2024-04-25T04:21:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2432 2024-04-25T04:21:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2431 2024-04-25T04:21:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2434 2024-04-25T04:21:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2433 2024-04-25T04:21:18.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:19.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2436 2024-04-25T04:21:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2437 2024-04-25T04:21:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2438 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2440 2024-04-25T04:21:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2439 2024-04-25T04:21:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2442 2024-04-25T04:21:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2443 2024-04-25T04:21:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2441 2024-04-25T04:21:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2444 2024-04-25T04:21:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2445 2024-04-25T04:21:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:23.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2448 2024-04-25T04:21:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2447 2024-04-25T04:21:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2446 2024-04-25T04:21:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:25.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2450 2024-04-25T04:21:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2449 2024-04-25T04:21:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2451 2024-04-25T04:21:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2452 2024-04-25T04:21:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:26.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2453 2024-04-25T04:21:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2455 2024-04-25T04:21:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2454 2024-04-25T04:21:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:27.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2458 2024-04-25T04:21:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2457 2024-04-25T04:21:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2456 2024-04-25T04:21:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:30.637 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:21:30.630+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667ab9c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:21:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2460 2024-04-25T04:21:31.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2459 2024-04-25T04:21:31.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:31.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:21:31.657+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665985000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:21:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2462 2024-04-25T04:21:32.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2461 2024-04-25T04:21:32.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2463 2024-04-25T04:21:32.593 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:21:32.587+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657f2400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:21:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2465 2024-04-25T04:21:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2464 2024-04-25T04:21:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2466 2024-04-25T04:21:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2468 2024-04-25T04:21:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2467 2024-04-25T04:21:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2469 2024-04-25T04:21:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2470 2024-04-25T04:21:35.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2473 2024-04-25T04:21:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2471 2024-04-25T04:21:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2472 2024-04-25T04:21:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2475 2024-04-25T04:21:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2474 2024-04-25T04:21:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2479 2024-04-25T04:21:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:39.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:39.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2476 2024-04-25T04:21:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2477 2024-04-25T04:21:40.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2478 2024-04-25T04:21:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2481 2024-04-25T04:21:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2480 2024-04-25T04:21:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:41.664 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:21:41.658+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667ac1400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:21:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2484 2024-04-25T04:21:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2482 2024-04-25T04:21:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2483 2024-04-25T04:21:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2485 2024-04-25T04:21:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:43.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2486 2024-04-25T04:21:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:44.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2487 2024-04-25T04:21:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2490 2024-04-25T04:21:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2489 2024-04-25T04:21:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2488 2024-04-25T04:21:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2492 2024-04-25T04:21:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2491 2024-04-25T04:21:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:46.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2493 2024-04-25T04:21:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2494 2024-04-25T04:21:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2496 2024-04-25T04:21:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2495 2024-04-25T04:21:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2499 2024-04-25T04:21:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2498 2024-04-25T04:21:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2500 2024-04-25T04:21:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2497 2024-04-25T04:21:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2502 2024-04-25T04:21:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2501 2024-04-25T04:21:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2503 2024-04-25T04:21:51.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2504 2024-04-25T04:21:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2506 2024-04-25T04:21:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:21:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:21:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:21:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2505 2024-04-25T04:21:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2510 2024-04-25T04:21:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2509 2024-04-25T04:21:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2508 2024-04-25T04:21:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:54.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:21:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:21:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:21:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2511 2024-04-25T04:21:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2513 2024-04-25T04:21:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2507 2024-04-25T04:21:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2514 2024-04-25T04:21:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:57.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2515 2024-04-25T04:21:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:21:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:21:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:21:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:21:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:21:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:21:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:21:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:21:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:21:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:21:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2512 2024-04-25T04:21:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:21:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:21:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:21:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2516 2024-04-25T04:21:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2518 2024-04-25T04:21:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2519 2024-04-25T04:21:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:21:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:21:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:21:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:21:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:21:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:21:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:21:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:21:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:21:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:21:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:21:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:21:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:21:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:00.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2520 2024-04-25T04:22:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2517 2024-04-25T04:22:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2523 2024-04-25T04:22:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2522 2024-04-25T04:22:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2521 2024-04-25T04:22:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2524 2024-04-25T04:22:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2525 2024-04-25T04:22:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:02.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2527 2024-04-25T04:22:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2528 2024-04-25T04:22:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2526 2024-04-25T04:22:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:04.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:04.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2529 2024-04-25T04:22:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2530 2024-04-25T04:22:05.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2533 2024-04-25T04:22:06.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2531 2024-04-25T04:22:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2532 2024-04-25T04:22:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2534 2024-04-25T04:22:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2535 2024-04-25T04:22:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:08.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:08.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2538 2024-04-25T04:22:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2537 2024-04-25T04:22:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2536 2024-04-25T04:22:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2540 2024-04-25T04:22:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2539 2024-04-25T04:22:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2541 2024-04-25T04:22:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2543 2024-04-25T04:22:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2542 2024-04-25T04:22:11.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:12.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:12.580 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:12.574+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596675dc400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2544 2024-04-25T04:22:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2545 2024-04-25T04:22:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:13.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:14.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2547 2024-04-25T04:22:14.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2546 2024-04-25T04:22:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:14.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2548 2024-04-25T04:22:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2549 2024-04-25T04:22:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2550 2024-04-25T04:22:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:16.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2551 2024-04-25T04:22:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2553 2024-04-25T04:22:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2552 2024-04-25T04:22:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:17.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2555 2024-04-25T04:22:17.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2554 2024-04-25T04:22:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2556 2024-04-25T04:22:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2558 2024-04-25T04:22:20.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2557 2024-04-25T04:22:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:20.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:20.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2559 2024-04-25T04:22:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:20.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2560 2024-04-25T04:22:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:21.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2561 2024-04-25T04:22:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2563 2024-04-25T04:22:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2562 2024-04-25T04:22:21.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2564 2024-04-25T04:22:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2565 2024-04-25T04:22:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:23.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:23.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:23.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2568 2024-04-25T04:22:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2567 2024-04-25T04:22:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2566 2024-04-25T04:22:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:24.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2570 2024-04-25T04:22:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2569 2024-04-25T04:22:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2573 2024-04-25T04:22:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2571 2024-04-25T04:22:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2572 2024-04-25T04:22:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2574 2024-04-25T04:22:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2575 2024-04-25T04:22:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:27.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:27.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:27.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:29.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:29.590 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:29.585+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596683ef000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:29.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:29.590+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667ab8c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:29.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:29.594+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657e6400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2577 2024-04-25T04:22:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2576 2024-04-25T04:22:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2578 2024-04-25T04:22:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2579 2024-04-25T04:22:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2580 2024-04-25T04:22:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:31.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:31.594+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596677d2400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:31.601 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:31.595+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667a82400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:31.608 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:31.602+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559664842400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:32.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2583 2024-04-25T04:22:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2581 2024-04-25T04:22:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2582 2024-04-25T04:22:32.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2584 2024-04-25T04:22:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2585 2024-04-25T04:22:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:33.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:34.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2586 2024-04-25T04:22:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2588 2024-04-25T04:22:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:34.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2587 2024-04-25T04:22:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:35.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:35.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:35.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2590 2024-04-25T04:22:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2589 2024-04-25T04:22:36.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:36.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2592 2024-04-25T04:22:36.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2593 2024-04-25T04:22:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2591 2024-04-25T04:22:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2594 2024-04-25T04:22:37.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2595 2024-04-25T04:22:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:38.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2597 2024-04-25T04:22:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2598 2024-04-25T04:22:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2596 2024-04-25T04:22:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2602 2024-04-25T04:22:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2601 2024-04-25T04:22:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2600 2024-04-25T04:22:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2599 2024-04-25T04:22:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2603 2024-04-25T04:22:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:22:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:22:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:42.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:22:42.620+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667aba800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:22:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2608 2024-04-25T04:22:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2607 2024-04-25T04:22:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2606 2024-04-25T04:22:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:42.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2604 2024-04-25T04:22:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2605 2024-04-25T04:22:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:22:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:22:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:22:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:22:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2609 2024-04-25T04:22:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2613 2024-04-25T04:22:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2611 2024-04-25T04:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:45.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:22:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:22:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:22:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 125745 2024-04-25T04:22:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:22:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:22:46.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:22:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2612 2024-04-25T04:22:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2610 2024-04-25T04:22:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2614 2024-04-25T04:22:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2615 2024-04-25T04:22:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2617 2024-04-25T04:22:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2616 2024-04-25T04:22:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:22:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:22:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2618 2024-04-25T04:22:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2619 2024-04-25T04:22:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 125879 2024-04-25T04:22:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126013 2024-04-25T04:22:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:22:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:22:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:22:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:22:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:22:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:22:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126168 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126344 2024-04-25T04:22:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2620 2024-04-25T04:22:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2621 2024-04-25T04:22:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2622 2024-04-25T04:22:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:22:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126499 2024-04-25T04:22:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126654 2024-04-25T04:22:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126829 2024-04-25T04:22:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:22:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 127005 2024-04-25T04:22:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:22:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:22:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 125612 2024-04-25T04:25:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 125746 2024-04-25T04:25:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 125880 2024-04-25T04:25:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126014 2024-04-25T04:25:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126169 2024-04-25T04:25:27.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126345 2024-04-25T04:25:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126500 2024-04-25T04:25:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:30.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126655 2024-04-25T04:25:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 126830 2024-04-25T04:25:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:25:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 127006 2024-04-25T04:25:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:25:33.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:25:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:25:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T04:25:33.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T04:25:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T04:25:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T04:25:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T04:25:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T04:25:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T04:25:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T04:25:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T04:25:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T04:25:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T04:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T04:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T04:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T04:25:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T04:25:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:25:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:25:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:25:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T04:25:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132977 2024-04-25T04:25:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T04:25:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T04:25:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T04:25:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T04:25:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T04:25:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T04:25:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T04:25:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T04:25:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:25:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T04:25:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:25:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:25:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:25:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:25:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:25:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:25:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:25:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:25:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:25:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:25:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:25:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=6083f9d9671363361a5e92030c06016e 2024-04-25T04:26:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T04:26:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=796daba08aa7aa56ddb2c05171d5b336 2024-04-25T04:26:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T04:26:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=8e558ccf343ffab2d7607dc0e5d9e2e3 2024-04-25T04:26:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T04:26:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T04:26:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T04:26:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T04:26:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T04:26:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T04:26:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T04:26:30.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=65ce59e20c98b76f837be0b92112e78f 2024-04-25T04:26:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T04:26:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=72f2a0080acefef7a9a2f94a41adb491 2024-04-25T04:26:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T04:26:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T04:26:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T04:26:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T04:26:30.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T04:26:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=3370eed69858c1c32ecb8c29caed846c 2024-04-25T04:26:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T04:26:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T04:26:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T04:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=36f9ce641c53a98a8c51daa3bdb5db75 2024-04-25T04:26:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T04:26:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=3419c3e9c812634621f0b5b50c0c7efc 2024-04-25T04:26:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T04:26:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T04:26:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T04:26:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T04:26:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T04:26:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=17d98d69b1c24fe07e49747584dc4d94 2024-04-25T04:26:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:31.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:26:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T04:26:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T04:26:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T04:26:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=b39413fdd41514df514627b29fb964f9 2024-04-25T04:26:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T04:26:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:26:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:26:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T04:26:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T04:26:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:26:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:26:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:26:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T04:26:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:32.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:26:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 33c83061-ffe0-4fdc-83ce-fadcd2db3404 2024-04-25T04:26:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:26:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 418dbbe0-c64c-4fab-89e3-76cc0e609f86 2024-04-25T04:26:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:32.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:32.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:26:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:33.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:26:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:26:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:33.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:26:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:26:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: feaa7574-7ded-461d-b6e2-1af898d5e863 2024-04-25T04:26:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:26:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 407d8be9-f19a-4331-91de-8e896ef24293 2024-04-25T04:26:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:33.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:26:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:26:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:33.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:26:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:26:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 33c83061-ffe0-4fdc-83ce-fadcd2db3404 2024-04-25T04:26:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:26:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 418dbbe0-c64c-4fab-89e3-76cc0e609f86 2024-04-25T04:26:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:34.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:26:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d7db7b9-4ab5-49e3-a40a-19ece53720ab 2024-04-25T04:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:35.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:26:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:26:35.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:26:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:26:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: feaa7574-7ded-461d-b6e2-1af898d5e863 2024-04-25T04:26:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:26:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 407d8be9-f19a-4331-91de-8e896ef24293 2024-04-25T04:26:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:26:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:26:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b04734ef-e82a-406f-aabe-120b4b0d23e6 2024-04-25T04:26:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:26:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 937679fc-49d1-4466-96ec-80fcc44b3c02 2024-04-25T04:26:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce19b877-ee6f-4303-8695-7d447d2f80f1 2024-04-25T04:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:36.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da170bd5-4439-490d-9f61-ca4a7eeacd5f 2024-04-25T04:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:26:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:26:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:26:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:26:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d7db7b9-4ab5-49e3-a40a-19ece53720ab 2024-04-25T04:26:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:37.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:26:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:26:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:26:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:26:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 937679fc-49d1-4466-96ec-80fcc44b3c02 2024-04-25T04:26:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b04734ef-e82a-406f-aabe-120b4b0d23e6 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce19b877-ee6f-4303-8695-7d447d2f80f1 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:26:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:26:38.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:38.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:26:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 33c83061-ffe0-4fdc-83ce-fadcd2db3404 2024-04-25T04:26:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:26:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 418dbbe0-c64c-4fab-89e3-76cc0e609f86 2024-04-25T04:26:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:26:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:26:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da170bd5-4439-490d-9f61-ca4a7eeacd5f 2024-04-25T04:26:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:26:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:26:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:26:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:26:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: feaa7574-7ded-461d-b6e2-1af898d5e863 2024-04-25T04:26:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:26:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 407d8be9-f19a-4331-91de-8e896ef24293 2024-04-25T04:26:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:26:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:26:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d7db7b9-4ab5-49e3-a40a-19ece53720ab 2024-04-25T04:26:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:42.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:26:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:26:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b04734ef-e82a-406f-aabe-120b4b0d23e6 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 937679fc-49d1-4466-96ec-80fcc44b3c02 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:26:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce19b877-ee6f-4303-8695-7d447d2f80f1 2024-04-25T04:26:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:26:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:43.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:26:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:26:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da170bd5-4439-490d-9f61-ca4a7eeacd5f 2024-04-25T04:26:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:26:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:26:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:26:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:26:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:26:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:26:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:16' 2024-04-25T04:26:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:26:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:26:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:26:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:26:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:26:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:26:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 33c83061-ffe0-4fdc-83ce-fadcd2db3404 2024-04-25T04:26:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 418dbbe0-c64c-4fab-89e3-76cc0e609f86 2024-04-25T04:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:47.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:47.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:47.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:26:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:26:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:26:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:26:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:26:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: feaa7574-7ded-461d-b6e2-1af898d5e863 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 407d8be9-f19a-4331-91de-8e896ef24293 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:26:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:26:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:26:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:26:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T04:26:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T04:26:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:26:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 33c83061-ffe0-4fdc-83ce-fadcd2db3404 2024-04-25T04:26:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 418dbbe0-c64c-4fab-89e3-76cc0e609f86 2024-04-25T04:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:26:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:26:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T04:26:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:26:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T04:26:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:48.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:26:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:26:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T04:26:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T04:26:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T04:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T04:26:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T04:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:26:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:26:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T04:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T04:26:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:26:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 407d8be9-f19a-4331-91de-8e896ef24293 2024-04-25T04:26:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:26:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: feaa7574-7ded-461d-b6e2-1af898d5e863 2024-04-25T04:26:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:49.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:26:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:26:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:26:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:26:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:49.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T04:26:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T04:26:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T04:26:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T04:26:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:26:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:26:50.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T04:26:50.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:26:50.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T04:26:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:26:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:26:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d7db7b9-4ab5-49e3-a40a-19ece53720ab 2024-04-25T04:26:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:50.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:26:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:26:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T04:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:26:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:26:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:26:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 937679fc-49d1-4466-96ec-80fcc44b3c02 2024-04-25T04:26:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:26:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b04734ef-e82a-406f-aabe-120b4b0d23e6 2024-04-25T04:26:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:26:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce19b877-ee6f-4303-8695-7d447d2f80f1 2024-04-25T04:26:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:51.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:51.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:51.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:26:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:26:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T04:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T04:26:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.335 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:26:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:26:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:26:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:26:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T04:26:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:26:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d7db7b9-4ab5-49e3-a40a-19ece53720ab 2024-04-25T04:26:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:26:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:26:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T04:26:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:26:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T04:26:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:51.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T04:26:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:26:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:26:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T04:26:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:26:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:52.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:26:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da170bd5-4439-490d-9f61-ca4a7eeacd5f 2024-04-25T04:26:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:26:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:26:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T04:26:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T04:26:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T04:26:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:52.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T04:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T04:26:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T04:26:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T04:26:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:26:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 937679fc-49d1-4466-96ec-80fcc44b3c02 2024-04-25T04:26:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:52.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:26:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce19b877-ee6f-4303-8695-7d447d2f80f1 2024-04-25T04:26:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:26:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b04734ef-e82a-406f-aabe-120b4b0d23e6 2024-04-25T04:26:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:52.561 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:26:52.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:26:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T04:26:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:26:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T04:26:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:52.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:26:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:26:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T04:26:52.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:26:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:26:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:52.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T04:26:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:26:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T04:26:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T04:26:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:26:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da170bd5-4439-490d-9f61-ca4a7eeacd5f 2024-04-25T04:26:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:53.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:53.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:53.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:53.501 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:26:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:26:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:53.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T04:26:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:53.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T04:26:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T04:26:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T04:26:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T04:26:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:26:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:26:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:26:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T04:26:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:26:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T04:26:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T04:26:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:26:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:26:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T04:26:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:26:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:26:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:54.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:54.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:26:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:26:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T04:26:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:26:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:26:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T04:26:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:26:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:26:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T04:26:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:26:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:26:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:26:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:26:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:26:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:26:46' 2024-04-25T04:26:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:26:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:26:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:26:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:26:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:26:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:26:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:26:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:26:56.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:26:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 04:26:46' 2024-04-25T04:26:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:26:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:26:46' 2024-04-25T04:26:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:26:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:26:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:26:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:26:46' 2024-04-25T04:26:56.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:26:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:26:56.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:26:46' 2024-04-25T04:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:26:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:26:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:26:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:26:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:26:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T04:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:26:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:26:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:26:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:26:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T04:26:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:26:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:26:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:26:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T04:26:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T04:26:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:26:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:27:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=3370eed69858c1c32ecb8c29caed846c 2024-04-25T04:27:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T04:27:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 3370eed69858c1c32ecb8c29caed846c != \3\3\7\0\e\e\d\6\9\8\5\8\c\1\c\3\2\e\c\b\8\c\2\9\c\a\e\d\8\4\6\c ]] 2024-04-25T04:27:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=8e558ccf343ffab2d7607dc0e5d9e2e3 2024-04-25T04:27:42.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T04:27:43.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 8e558ccf343ffab2d7607dc0e5d9e2e3 != \8\e\5\5\8\c\c\f\3\4\3\f\f\a\b\2\d\7\6\0\7\d\c\0\e\5\d\9\e\2\e\3 ]] 2024-04-25T04:27:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=36f9ce641c53a98a8c51daa3bdb5db75 2024-04-25T04:27:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T04:27:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 36f9ce641c53a98a8c51daa3bdb5db75 != \3\6\f\9\c\e\6\4\1\c\5\3\a\9\8\a\8\c\5\1\d\a\a\3\b\d\b\5\d\b\7\5 ]] 2024-04-25T04:27:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=796daba08aa7aa56ddb2c05171d5b336 2024-04-25T04:27:46.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T04:27:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 796daba08aa7aa56ddb2c05171d5b336 != \7\9\6\d\a\b\a\0\8\a\a\7\a\a\5\6\d\d\b\2\c\0\5\1\7\1\d\5\b\3\3\6 ]] 2024-04-25T04:27:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=72f2a0080acefef7a9a2f94a41adb491 2024-04-25T04:27:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T04:27:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 72f2a0080acefef7a9a2f94a41adb491 != \7\2\f\2\a\0\0\8\0\a\c\e\f\e\f\7\a\9\a\2\f\9\4\a\4\1\a\d\b\4\9\1 ]] 2024-04-25T04:27:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=17d98d69b1c24fe07e49747584dc4d94 2024-04-25T04:27:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T04:27:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=65ce59e20c98b76f837be0b92112e78f 2024-04-25T04:27:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T04:27:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 17d98d69b1c24fe07e49747584dc4d94 != \1\7\d\9\8\d\6\9\b\1\c\2\4\f\e\0\7\e\4\9\7\4\7\5\8\4\d\c\4\d\9\4 ]] 2024-04-25T04:27:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 65ce59e20c98b76f837be0b92112e78f != \6\5\c\e\5\9\e\2\0\c\9\8\b\7\6\f\8\3\7\b\e\0\b\9\2\1\1\2\e\7\8\f ]] 2024-04-25T04:27:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=3419c3e9c812634621f0b5b50c0c7efc 2024-04-25T04:27:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T04:27:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=b39413fdd41514df514627b29fb964f9 2024-04-25T04:27:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T04:27:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 3419c3e9c812634621f0b5b50c0c7efc != \3\4\1\9\c\3\e\9\c\8\1\2\6\3\4\6\2\1\f\0\b\5\b\5\0\c\0\c\7\e\f\c ]] 2024-04-25T04:27:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ b39413fdd41514df514627b29fb964f9 != \b\3\9\4\1\3\f\d\d\4\1\5\1\4\d\f\5\1\4\6\2\7\b\2\9\f\b\9\6\4\f\9 ]] 2024-04-25T04:27:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132978 2024-04-25T04:27:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=6083f9d9671363361a5e92030c06016e 2024-04-25T04:27:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T04:27:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 6083f9d9671363361a5e92030c06016e != \6\0\8\3\f\9\d\9\6\7\1\3\6\3\3\6\1\a\5\e\9\2\0\3\0\c\0\6\0\1\6\e ]] 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132980 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132982 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132984 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132986 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132988 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132990 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132993 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 132994 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:27:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:27:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:27:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:27:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:27:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T04:27:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:27:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:27:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:27:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:27:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:27:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:27:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ce19b877-ee6f-4303-8695-7d447d2f80f1 2024-04-25T04:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3407667.2,"bytes_per_snapshot":68153344.0,"last_snapshot_bytes":68153344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:27:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:27:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:27:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:27:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3407667.2,"bytes_per_snapshot":68153344.0,"last_snapshot_bytes":68153344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:27:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:27:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:27:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:27:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:27:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:27:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:27:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:27:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:27:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:27:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:27:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:27:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:27:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:27:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:27:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:27:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1ce19b877-ee6f-4303-8695-7d447d2f80f1up+replayingreplaying, {"bytes_per_second":3407667.2,"bytes_per_snapshot":68153344.0,"last_snapshot_bytes":68153344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+replayingreplaying, {"bytes_per_second":3162931.2,"bytes_per_snapshot":63258624.0,"last_snapshot_bytes":63258624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019216,"remote_snapshot_timestamp":1714019216,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+replayingreplaying, {"bytes_per_second":3637452.8,"bytes_per_snapshot":72749056.0,"last_snapshot_bytes":72749056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019211,"remote_snapshot_timestamp":1714019211,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+replayingreplaying, {"bytes_per_second":3599769.6,"bytes_per_snapshot":71995392.0,"last_snapshot_bytes":71995392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+replayingreplaying, {"bytes_per_second":3686400.0,"bytes_per_snapshot":73728000.0,"last_snapshot_bytes":73728000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+replayingreplaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+replayingreplaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:27:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:27:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:27:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:27:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:27:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:27:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:27:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:27:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:27:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T04:27:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:27:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:27:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T04:27:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T04:27:52.823 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:27:52.817+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664563860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:27:52.830 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:27:52.817+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508c80 [3/ce19b877-ee6f-4303-8695-7d447d2f80f1] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:27:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:27:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T04:27:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:27:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:27:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:27:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T04:27:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T04:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T04:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:27:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:27:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T04:27:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:27:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:27:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:27:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:27:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:27:55.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:27:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:27:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: fd7ddf4a-86de-4368-a54f-9f406753574c 2024-04-25T04:27:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:27:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3162931.2,"bytes_per_snapshot":63258624.0,"last_snapshot_bytes":63258624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019216,"remote_snapshot_timestamp":1714019216,"replay_state":"idle"} 2024-04-25T04:27:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:27:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:27:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:27:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:27:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:27:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3162931.2,"bytes_per_snapshot":63258624.0,"last_snapshot_bytes":63258624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019216,"remote_snapshot_timestamp":1714019216,"replay_state":"idle"} 2024-04-25T04:27:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:27:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:27:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:27:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:27:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:27:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:27:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:27:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:27:55.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:27:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:27:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary2fd7ddf4a-86de-4368-a54f-9f406753574cup+replayingreplaying, {"bytes_per_second":3162931.2,"bytes_per_snapshot":63258624.0,"last_snapshot_bytes":63258624,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019216,"remote_snapshot_timestamp":1714019216,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+replayingreplaying, {"bytes_per_second":3637452.8,"bytes_per_snapshot":72749056.0,"last_snapshot_bytes":72749056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019211,"remote_snapshot_timestamp":1714019211,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+replayingreplaying, {"bytes_per_second":3599769.6,"bytes_per_snapshot":71995392.0,"last_snapshot_bytes":71995392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+replayingreplaying, {"bytes_per_second":3686400.0,"bytes_per_snapshot":73728000.0,"last_snapshot_bytes":73728000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+replayingreplaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+replayingreplaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:27:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:27:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:27:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:27:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:27:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:27:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:27:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:27:55.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:27:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T04:27:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:27:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:27:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T04:27:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T04:27:55.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:27:55.643+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668483d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:27:55.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:27:55.643+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/fd7ddf4a-86de-4368-a54f-9f406753574c] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:27:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:27:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T04:27:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:27:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:27:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:27:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T04:27:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T04:27:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:27:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:27:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T04:27:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:27:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:27:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T04:27:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:27:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:27:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:27:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:27:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:27:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:27:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:27:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1d7db7b9-4ab5-49e3-a40a-19ece53720ab 2024-04-25T04:27:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:27:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3637452.8,"bytes_per_snapshot":72749056.0,"last_snapshot_bytes":72749056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019211,"remote_snapshot_timestamp":1714019211,"replay_state":"idle"} 2024-04-25T04:27:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:27:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:27:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:27:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:27:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:27:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3637452.8,"bytes_per_snapshot":72749056.0,"last_snapshot_bytes":72749056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019211,"remote_snapshot_timestamp":1714019211,"replay_state":"idle"} 2024-04-25T04:27:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:27:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:27:58.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:27:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:27:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:27:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:27:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:27:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:27:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:27:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:27:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary31d7db7b9-4ab5-49e3-a40a-19ece53720abup+replayingreplaying, {"bytes_per_second":3637452.8,"bytes_per_snapshot":72749056.0,"last_snapshot_bytes":72749056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019211,"remote_snapshot_timestamp":1714019211,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+replayingreplaying, {"bytes_per_second":3599769.6,"bytes_per_snapshot":71995392.0,"last_snapshot_bytes":71995392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+replayingreplaying, {"bytes_per_second":3686400.0,"bytes_per_snapshot":73728000.0,"last_snapshot_bytes":73728000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+replayingreplaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+replayingreplaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:27:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:27:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:27:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:27:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:27:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:27:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:27:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:27:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:27:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:27:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T04:27:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:27:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:27:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:27:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T04:27:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T04:27:58.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:27:58.657+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559668602340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:27:58.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:27:58.657+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/1d7db7b9-4ab5-49e3-a40a-19ece53720ab] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T04:28:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T04:28:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:28:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:28:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:28:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 33c83061-ffe0-4fdc-83ce-fadcd2db3404 2024-04-25T04:28:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:28:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3599769.6,"bytes_per_snapshot":71995392.0,"last_snapshot_bytes":71995392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"} 2024-04-25T04:28:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:28:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:28:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:28:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:28:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3599769.6,"bytes_per_snapshot":71995392.0,"last_snapshot_bytes":71995392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"} 2024-04-25T04:28:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:28:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:28:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:01.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:28:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:28:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:28:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:28:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:28:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary433c83061-ffe0-4fdc-83ce-fadcd2db3404up+replayingreplaying, {"bytes_per_second":3599769.6,"bytes_per_snapshot":71995392.0,"last_snapshot_bytes":71995392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+replayingreplaying, {"bytes_per_second":3686400.0,"bytes_per_snapshot":73728000.0,"last_snapshot_bytes":73728000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+replayingreplaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+replayingreplaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:28:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:28:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:28:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:28:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:28:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:28:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T04:28:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:28:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T04:28:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T04:28:01.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:01.808+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596654f0820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:28:01.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:01.808+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/33c83061-ffe0-4fdc-83ce-fadcd2db3404] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T04:28:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T04:28:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:28:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T04:28:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:03.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:28:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:28:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: b04734ef-e82a-406f-aabe-120b4b0d23e6 2024-04-25T04:28:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:28:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3686400.0,"bytes_per_snapshot":73728000.0,"last_snapshot_bytes":73728000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:28:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:28:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:28:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:28:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:28:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:28:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3686400.0,"bytes_per_snapshot":73728000.0,"last_snapshot_bytes":73728000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:28:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:28:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:28:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:28:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:28:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:28:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:28:03.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:03.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:28:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary5b04734ef-e82a-406f-aabe-120b4b0d23e6up+replayingreplaying, {"bytes_per_second":3686400.0,"bytes_per_snapshot":73728000.0,"last_snapshot_bytes":73728000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+replayingreplaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+replayingreplaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:28:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:28:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:28:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:28:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:28:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:28:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T04:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T04:28:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T04:28:04.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:04.142+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667ae3ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:28:04.148 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:04.142+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/b04734ef-e82a-406f-aabe-120b4b0d23e6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T04:28:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T04:28:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:28:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T04:28:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:28:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:28:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 407d8be9-f19a-4331-91de-8e896ef24293 2024-04-25T04:28:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:28:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"} 2024-04-25T04:28:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:28:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:28:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:28:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:28:07.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:28:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"} 2024-04-25T04:28:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:28:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:28:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:28:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:28:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:28:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:28:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:28:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:28:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary6407d8be9-f19a-4331-91de-8e896ef24293up+replayingreplaying, {"bytes_per_second":3233382.4,"bytes_per_snapshot":64667648.0,"last_snapshot_bytes":64667648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+replayingreplaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:28:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:28:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:28:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:28:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:28:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:28:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T04:28:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:28:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T04:28:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T04:28:07.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:07.707+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664f0a1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:28:07.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:07.707+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/407d8be9-f19a-4331-91de-8e896ef24293] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T04:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T04:28:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T04:28:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:28:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T04:28:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:28:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: da170bd5-4439-490d-9f61-ca4a7eeacd5f 2024-04-25T04:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:28:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:28:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:28:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:28:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:28:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:28:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:28:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:28:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:28:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:28:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:28:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:28:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary7da170bd5-4439-490d-9f61-ca4a7eeacd5fup+replayingreplaying, {"bytes_per_second":3308339.2,"bytes_per_snapshot":66166784.0,"last_snapshot_bytes":66166784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:28:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:28:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:28:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:28:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:28:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:28:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T04:28:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:28:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T04:28:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T04:28:11.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:11.128+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667bc6b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:28:11.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:11.128+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508a00 [3/da170bd5-4439-490d-9f61-ca4a7eeacd5f] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T04:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T04:28:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:12.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:28:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:28:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:28:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 937679fc-49d1-4466-96ec-80fcc44b3c02 2024-04-25T04:28:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:28:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:28:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:28:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:27:46' 2024-04-25T04:28:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:28:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:28:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:28:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"} 2024-04-25T04:28:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:28:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:28:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:28:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:28:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:28:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 04:27:46' 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:27:46' 2024-04-25T04:28:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:28:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":3332915.2,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary8937679fc-49d1-4466-96ec-80fcc44b3c02up+replayingreplaying, {"bytes_per_second":3703808.0,"bytes_per_snapshot":74076160.0,"last_snapshot_bytes":74076160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019213,"remote_snapshot_timestamp":1714019213,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":3311411.2,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:27:46' 2024-04-25T04:28:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:28:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:28:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:27:46' 2024-04-25T04:28:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:28:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:28:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:28:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T04:28:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:28:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T04:28:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T04:28:14.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:14.133+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596679c7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:28:14.138 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:14.133+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/937679fc-49d1-4466-96ec-80fcc44b3c02] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T04:28:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T04:28:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T04:28:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:28:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T04:28:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:28:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:28:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 418dbbe0-c64c-4fab-89e3-76cc0e609f86 2024-04-25T04:28:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:28:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"} 2024-04-25T04:28:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:28:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:28:16' 2024-04-25T04:28:16.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:28:16.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:28:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:28:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"} 2024-04-25T04:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:16.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:28:16.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:28:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:28:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:28:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:28:16' 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 04:28:16' 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:28:16' 2024-04-25T04:28:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:28:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:28:16image-primary9418dbbe0-c64c-4fab-89e3-76cc0e609f86up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66228224.0,"last_snapshot_bytes":66228224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019208,"remote_snapshot_timestamp":1714019208,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:28:16' 2024-04-25T04:28:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:28:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:28:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:28:16' 2024-04-25T04:28:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:28:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:28:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:16.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:28:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T04:28:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:28:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T04:28:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T04:28:17.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:17.175+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664f04820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:28:17.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:17.175+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/418dbbe0-c64c-4fab-89e3-76cc0e609f86] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T04:28:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T04:28:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T04:28:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:28:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T04:28:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:28:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:28:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:28:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:28:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: feaa7574-7ded-461d-b6e2-1af898d5e863 2024-04-25T04:28:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:28:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"} 2024-04-25T04:28:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:28:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:28:16' 2024-04-25T04:28:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:28:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:28:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"} 2024-04-25T04:28:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:28:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:28:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:28:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:28:16' 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 04:28:16' 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:28:16' 2024-04-25T04:28:19.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:28:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:28:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:28:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary10feaa7574-7ded-461d-b6e2-1af898d5e863up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66658304.0,"last_snapshot_bytes":66658304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019209,"remote_snapshot_timestamp":1714019209,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:28:16' 2024-04-25T04:28:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:28:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:28:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:28:16' 2024-04-25T04:28:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:28:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:28:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:28:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:28:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:28:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T04:28:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:28:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T04:28:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T04:28:20.174 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:20.168+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667879520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:28:20.175 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:28:20.168+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb680 [3/feaa7574-7ded-461d-b6e2-1af898d5e863] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:28:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:28:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T04:28:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:28:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:28:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T04:28:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T04:28:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T04:28:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T04:28:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T04:28:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T04:28:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 37f2b30de43c 2024-04-25T04:28:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.37f2b30de43c 2024-04-25T04:28:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:21 2024 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:21 2024 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:21 2024 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: cd0e1482-d103-434e-b5f9-275b2b7ff625 2024-04-25T04:28:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T04:28:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T04:28:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T04:28:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 54e8a7fd-9ac3-4e17-ae77-9c9c9657fa11 2024-04-25T04:28:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:22.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T04:28:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T04:28:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:23.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T04:28:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T04:28:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T04:28:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T04:28:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T04:28:23.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2644 2024-04-25T04:28:23.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 38025fbe9797 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.38025fbe9797 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:23.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:23 2024 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:23 2024 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:23 2024 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:28:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:23.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T04:28:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T04:28:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T04:28:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: b1bc2981-fd8b-434f-9f86-b69348997fae 2024-04-25T04:28:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T04:28:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T04:28:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:24.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T04:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T04:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T04:28:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:28:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T04:28:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T04:28:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T04:28:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T04:28:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:24.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2646 2024-04-25T04:28:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T04:28:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T04:28:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 38119ff36147 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.38119ff36147 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:24 2024 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:24 2024 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:24 2024 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ab15d283-8f3b-4ab3-8239-581e191f477d 2024-04-25T04:28:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T04:28:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T04:28:24.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:24.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T04:28:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 911e85f1-0047-454c-9c51-478a3deac3e1 2024-04-25T04:28:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:24.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T04:28:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T04:28:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:25.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T04:28:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T04:28:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:25.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T04:28:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T04:28:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T04:28:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2648 2024-04-25T04:28:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3820458bf234 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3820458bf234 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:25 2024 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:25 2024 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:25 2024 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 3b4e86f7-dfe9-45c5-8e30-08fda287cc10 2024-04-25T04:28:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T04:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T04:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T04:28:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ca346690-e1ef-40a9-9502-1aeed28c52ba 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T04:28:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T04:28:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T04:28:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T04:28:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T04:28:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T04:28:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T04:28:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:26.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T04:28:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T04:28:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2650 2024-04-25T04:28:26.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T04:28:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 382ff42da13e 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.382ff42da13e 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:26 2024 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:26 2024 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:26 2024 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 4aa05b27-d3f3-4b02-9202-0be6094130ea 2024-04-25T04:28:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T04:28:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T04:28:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T04:28:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 3fb1d7be-fd93-4a0d-b9ec-ba3c6e91e574 2024-04-25T04:28:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T04:28:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T04:28:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T04:28:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T04:28:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T04:28:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2652 2024-04-25T04:28:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2653 2024-04-25T04:28:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T04:28:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2655 2024-04-25T04:28:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T04:28:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T04:28:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3840684a8683 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3840684a8683 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:27 2024 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:27 2024 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:27 2024 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 6f3801fd-9c66-4c6b-9c17-42356988b18b 2024-04-25T04:28:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T04:28:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T04:28:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T04:28:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: eef0aaed-e7ac-4e93-b03d-018981b55efc 2024-04-25T04:28:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T04:28:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T04:28:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:29.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T04:28:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T04:28:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2657 2024-04-25T04:28:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2656 2024-04-25T04:28:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T04:28:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3854678c70a2 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3854678c70a2 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:29 2024 2024-04-25T04:28:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:29 2024 2024-04-25T04:28:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:29 2024 2024-04-25T04:28:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:28:29.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T04:28:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T04:28:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T04:28:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: a1be740b-99d3-435c-ab1b-2daf43bdcc4d 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:29.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:29.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T04:28:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T04:28:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T04:28:30.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T04:28:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T04:28:30.437 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:30.438 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T04:28:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2660 2024-04-25T04:28:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:30.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2659 2024-04-25T04:28:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:30.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T04:28:30.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:30.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 386624ef7f36 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.386624ef7f36 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:30 2024 2024-04-25T04:28:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:30 2024 2024-04-25T04:28:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:30 2024 2024-04-25T04:28:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: a5edbcf7-797e-403b-a818-2107011039af 2024-04-25T04:28:30.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T04:28:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T04:28:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T04:28:30.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:31.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 02b17572-ab5d-46f9-96d2-4e198afc2452 2024-04-25T04:28:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:31.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T04:28:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T04:28:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:28:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T04:28:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T04:28:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:31.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T04:28:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T04:28:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:31.488 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T04:28:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:31.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2663 2024-04-25T04:28:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2664 2024-04-25T04:28:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2662 2024-04-25T04:28:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T04:28:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T04:28:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 387a5b52da2c 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.387a5b52da2c 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:31 2024 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:31 2024 2024-04-25T04:28:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:31 2024 2024-04-25T04:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 6818b672-2285-4be4-a1ae-7405485002e6 2024-04-25T04:28:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T04:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T04:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:31.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T04:28:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 0b0dc885-77fb-454d-974b-d8348a12685e 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T04:28:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T04:28:32.137 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T04:28:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T04:28:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T04:28:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T04:28:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T04:28:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:28:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T04:28:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T04:28:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:28:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2667 2024-04-25T04:28:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2666 2024-04-25T04:28:32.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T04:28:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 388c22e2390c 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.388c22e2390c 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:28:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:28:32 2024 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:28:32 2024 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:28:32 2024 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 9538dbfc-4e8b-447c-a5fa-3968992771aa 2024-04-25T04:28:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:28:32.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T04:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T04:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:28:33.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T04:28:33.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:28:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:28:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:28:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 00af2729-bcd6-4a6a-a6d6-37421fee2ee3 2024-04-25T04:28:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:28:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:28:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:28:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:28:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:28:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:28:33.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:28:33.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T04:28:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T04:28:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:28:33.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T04:28:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:28:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T04:28:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:33.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137234 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T04:28:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T04:28:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:28:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T04:28:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2670 2024-04-25T04:28:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2671 2024-04-25T04:28:33.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2669 2024-04-25T04:28:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:28:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:28:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:34.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2673 2024-04-25T04:28:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2672 2024-04-25T04:28:34.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:28:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:28:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2676 2024-04-25T04:28:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2675 2024-04-25T04:28:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2674 2024-04-25T04:28:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:28:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2678 2024-04-25T04:28:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2677 2024-04-25T04:28:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2679 2024-04-25T04:28:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2681 2024-04-25T04:28:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2680 2024-04-25T04:28:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2683 2024-04-25T04:28:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2682 2024-04-25T04:28:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:28:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2686 2024-04-25T04:28:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2684 2024-04-25T04:28:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2685 2024-04-25T04:28:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:28:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:40.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:28:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2687 2024-04-25T04:28:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:28:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:28:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:41.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:41.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2688 2024-04-25T04:28:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2690 2024-04-25T04:28:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2689 2024-04-25T04:28:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2691 2024-04-25T04:28:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:42.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:43.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:43.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2692 2024-04-25T04:28:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:28:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:43.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:28:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2693 2024-04-25T04:28:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:43.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2696 2024-04-25T04:28:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2694 2024-04-25T04:28:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:28:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:28:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:44.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2695 2024-04-25T04:28:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2697 2024-04-25T04:28:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:28:45.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:28:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:45.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2698 2024-04-25T04:28:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2699 2024-04-25T04:28:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2700 2024-04-25T04:28:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2701 2024-04-25T04:28:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2702 2024-04-25T04:28:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:28:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:28:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:46.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2703 2024-04-25T04:28:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:47.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:28:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2704 2024-04-25T04:28:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2705 2024-04-25T04:28:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:28:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:28:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2707 2024-04-25T04:28:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:49.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2706 2024-04-25T04:28:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2709 2024-04-25T04:28:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2708 2024-04-25T04:28:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2710 2024-04-25T04:28:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:50.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:28:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:28:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2711 2024-04-25T04:28:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2712 2024-04-25T04:28:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:28:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:28:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:28:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:28:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2714 2024-04-25T04:28:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2713 2024-04-25T04:28:53.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2716 2024-04-25T04:28:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2715 2024-04-25T04:28:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2717 2024-04-25T04:28:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2718 2024-04-25T04:28:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:28:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2719 2024-04-25T04:28:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2720 2024-04-25T04:28:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:28:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:28:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:28:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:28:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:28:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:28:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2721 2024-04-25T04:28:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:28:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:28:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:28:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:28:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:28:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2722 2024-04-25T04:28:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:28:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:28:57.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:28:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:28:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:28:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:28:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2724 2024-04-25T04:28:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2723 2024-04-25T04:28:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2725 2024-04-25T04:28:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2726 2024-04-25T04:28:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:28:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:28:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:28:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:28:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:28:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:28:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2728 2024-04-25T04:28:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2727 2024-04-25T04:28:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:28:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:28:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:28:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:28:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:28:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:28:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2729 2024-04-25T04:29:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2730 2024-04-25T04:29:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2731 2024-04-25T04:29:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2732 2024-04-25T04:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:02.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2733 2024-04-25T04:29:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2734 2024-04-25T04:29:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2736 2024-04-25T04:29:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2735 2024-04-25T04:29:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:03.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2738 2024-04-25T04:29:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2737 2024-04-25T04:29:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:06.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2739 2024-04-25T04:29:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2740 2024-04-25T04:29:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2742 2024-04-25T04:29:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:06.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2741 2024-04-25T04:29:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2744 2024-04-25T04:29:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:07.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2743 2024-04-25T04:29:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2746 2024-04-25T04:29:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2745 2024-04-25T04:29:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2748 2024-04-25T04:29:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2747 2024-04-25T04:29:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2750 2024-04-25T04:29:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2749 2024-04-25T04:29:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2752 2024-04-25T04:29:11.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2751 2024-04-25T04:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:12.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:12.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2753 2024-04-25T04:29:13.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2754 2024-04-25T04:29:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:13.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:13.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2756 2024-04-25T04:29:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2755 2024-04-25T04:29:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:14.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2758 2024-04-25T04:29:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2757 2024-04-25T04:29:15.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2759 2024-04-25T04:29:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2760 2024-04-25T04:29:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:16.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:16.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:16.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2761 2024-04-25T04:29:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2762 2024-04-25T04:29:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:17.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2764 2024-04-25T04:29:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2763 2024-04-25T04:29:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:18.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2766 2024-04-25T04:29:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:18.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2765 2024-04-25T04:29:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2768 2024-04-25T04:29:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:19.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2767 2024-04-25T04:29:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2770 2024-04-25T04:29:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2769 2024-04-25T04:29:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:20.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:21.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2772 2024-04-25T04:29:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2771 2024-04-25T04:29:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2774 2024-04-25T04:29:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:22.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2773 2024-04-25T04:29:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:23.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2775 2024-04-25T04:29:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2776 2024-04-25T04:29:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2778 2024-04-25T04:29:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2777 2024-04-25T04:29:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2779 2024-04-25T04:29:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:26.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2780 2024-04-25T04:29:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:26.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:26.785 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:29:26.779+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667566400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:29:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2781 2024-04-25T04:29:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2782 2024-04-25T04:29:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2785 2024-04-25T04:29:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2783 2024-04-25T04:29:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2786 2024-04-25T04:29:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2784 2024-04-25T04:29:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2788 2024-04-25T04:29:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2787 2024-04-25T04:29:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:29.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2790 2024-04-25T04:29:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2789 2024-04-25T04:29:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:31.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2792 2024-04-25T04:29:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:31.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2791 2024-04-25T04:29:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2794 2024-04-25T04:29:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2793 2024-04-25T04:29:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2796 2024-04-25T04:29:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2795 2024-04-25T04:29:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2798 2024-04-25T04:29:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2797 2024-04-25T04:29:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:35.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2801 2024-04-25T04:29:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2800 2024-04-25T04:29:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:36.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2799 2024-04-25T04:29:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2802 2024-04-25T04:29:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2805 2024-04-25T04:29:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2803 2024-04-25T04:29:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2804 2024-04-25T04:29:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2806 2024-04-25T04:29:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2808 2024-04-25T04:29:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2807 2024-04-25T04:29:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2809 2024-04-25T04:29:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2811 2024-04-25T04:29:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:41.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2810 2024-04-25T04:29:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2812 2024-04-25T04:29:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:42.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2814 2024-04-25T04:29:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2813 2024-04-25T04:29:43.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2815 2024-04-25T04:29:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2816 2024-04-25T04:29:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2819 2024-04-25T04:29:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:44.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2817 2024-04-25T04:29:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2818 2024-04-25T04:29:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:44.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2820 2024-04-25T04:29:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2821 2024-04-25T04:29:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2822 2024-04-25T04:29:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2824 2024-04-25T04:29:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2827 2024-04-25T04:29:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2823 2024-04-25T04:29:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2826 2024-04-25T04:29:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2825 2024-04-25T04:29:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2828 2024-04-25T04:29:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2829 2024-04-25T04:29:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:51.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2831 2024-04-25T04:29:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2830 2024-04-25T04:29:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:51.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2832 2024-04-25T04:29:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2836 2024-04-25T04:29:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2833 2024-04-25T04:29:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2837 2024-04-25T04:29:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2835 2024-04-25T04:29:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:54.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2838 2024-04-25T04:29:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2839 2024-04-25T04:29:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:54.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2840 2024-04-25T04:29:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2841 2024-04-25T04:29:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:29:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:29:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:29:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:29:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2834 2024-04-25T04:29:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2842 2024-04-25T04:29:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:29:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:29:56.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:29:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:29:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:29:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:29:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:29:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:29:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:29:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2843 2024-04-25T04:29:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:57.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2844 2024-04-25T04:29:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:29:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:29:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:29:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:29:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:29:57.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:29:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2846 2024-04-25T04:29:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2845 2024-04-25T04:29:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:29:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:29:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:29:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:29:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:29:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:29:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:29:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:29:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:29:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2848 2024-04-25T04:29:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2847 2024-04-25T04:29:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:29:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:29:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:29:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:29:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:29:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:29:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2850 2024-04-25T04:30:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2849 2024-04-25T04:30:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:00.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2852 2024-04-25T04:30:01.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2851 2024-04-25T04:30:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2854 2024-04-25T04:30:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2853 2024-04-25T04:30:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2856 2024-04-25T04:30:03.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2855 2024-04-25T04:30:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2858 2024-04-25T04:30:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2857 2024-04-25T04:30:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2859 2024-04-25T04:30:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2860 2024-04-25T04:30:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:05.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2861 2024-04-25T04:30:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2862 2024-04-25T04:30:06.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2864 2024-04-25T04:30:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:07.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:07.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:07.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:30:07.667+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559668a3ac00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:30:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2863 2024-04-25T04:30:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2866 2024-04-25T04:30:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2865 2024-04-25T04:30:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2867 2024-04-25T04:30:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2868 2024-04-25T04:30:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2869 2024-04-25T04:30:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2870 2024-04-25T04:30:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:11.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2871 2024-04-25T04:30:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2872 2024-04-25T04:30:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:12.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:30:12.642+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596675adc00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:30:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2875 2024-04-25T04:30:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:13.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2873 2024-04-25T04:30:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2876 2024-04-25T04:30:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2874 2024-04-25T04:30:13.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2878 2024-04-25T04:30:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2877 2024-04-25T04:30:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:14.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2879 2024-04-25T04:30:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2880 2024-04-25T04:30:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:16.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2881 2024-04-25T04:30:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2882 2024-04-25T04:30:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2883 2024-04-25T04:30:16.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:17.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2884 2024-04-25T04:30:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:18.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2885 2024-04-25T04:30:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2886 2024-04-25T04:30:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:18.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:18.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:19.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:30:19.710+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966575e800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:30:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2888 2024-04-25T04:30:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2887 2024-04-25T04:30:20.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:20.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:21.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2889 2024-04-25T04:30:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2890 2024-04-25T04:30:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2891 2024-04-25T04:30:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2892 2024-04-25T04:30:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2894 2024-04-25T04:30:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2893 2024-04-25T04:30:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2895 2024-04-25T04:30:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2896 2024-04-25T04:30:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:23.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2897 2024-04-25T04:30:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2898 2024-04-25T04:30:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2900 2024-04-25T04:30:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2899 2024-04-25T04:30:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:26.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:26.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:26.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2902 2024-04-25T04:30:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2901 2024-04-25T04:30:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2904 2024-04-25T04:30:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2903 2024-04-25T04:30:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2905 2024-04-25T04:30:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2906 2024-04-25T04:30:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2907 2024-04-25T04:30:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:28.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:28.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:28.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2908 2024-04-25T04:30:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2909 2024-04-25T04:30:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2910 2024-04-25T04:30:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2912 2024-04-25T04:30:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2911 2024-04-25T04:30:31.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:32.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2913 2024-04-25T04:30:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2914 2024-04-25T04:30:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:32.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2917 2024-04-25T04:30:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2915 2024-04-25T04:30:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2916 2024-04-25T04:30:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2918 2024-04-25T04:30:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2919 2024-04-25T04:30:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2920 2024-04-25T04:30:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:36.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2921 2024-04-25T04:30:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2922 2024-04-25T04:30:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2925 2024-04-25T04:30:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:37.666 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:30:37.662+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665a5a000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:30:37.669 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:30:37.665+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596659ab800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:30:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2924 2024-04-25T04:30:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2923 2024-04-25T04:30:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2926 2024-04-25T04:30:38.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2927 2024-04-25T04:30:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2928 2024-04-25T04:30:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:30:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:30:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:30:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2930 2024-04-25T04:30:40.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2929 2024-04-25T04:30:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2933 2024-04-25T04:30:41.620 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:41.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:41.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2932 2024-04-25T04:30:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2931 2024-04-25T04:30:41.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:42.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2935 2024-04-25T04:30:42.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2934 2024-04-25T04:30:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:42.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2937 2024-04-25T04:30:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2936 2024-04-25T04:30:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:30:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137368 2024-04-25T04:30:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:30:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:30:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:30:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2938 2024-04-25T04:30:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2939 2024-04-25T04:30:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2940 2024-04-25T04:30:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:30:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:30:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:30:46.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2941 2024-04-25T04:30:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2942 2024-04-25T04:30:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2943 2024-04-25T04:30:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2944 2024-04-25T04:30:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2945 2024-04-25T04:30:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2946 2024-04-25T04:30:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:30:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137502 2024-04-25T04:30:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:30:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:30:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2947 2024-04-25T04:30:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:30:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:30:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:30:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137636 2024-04-25T04:30:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:30:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:30:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:30:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137790 2024-04-25T04:30:50.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:30:50.717+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667961800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:30:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2948 2024-04-25T04:30:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:30:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:30:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:30:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137966 2024-04-25T04:30:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:51.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:30:51.722+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657cb800 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:30:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2949 2024-04-25T04:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:30:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:30:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2950 2024-04-25T04:30:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2951 2024-04-25T04:30:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2952 2024-04-25T04:30:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:30:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138121 2024-04-25T04:30:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138296 2024-04-25T04:30:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138452 2024-04-25T04:30:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:30:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138627 2024-04-25T04:30:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:30:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:30:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137235 2024-04-25T04:33:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137369 2024-04-25T04:33:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137503 2024-04-25T04:33:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137637 2024-04-25T04:33:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137791 2024-04-25T04:33:27.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:27.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:27.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:27.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 137967 2024-04-25T04:33:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138122 2024-04-25T04:33:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138297 2024-04-25T04:33:31.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138453 2024-04-25T04:33:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:33:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 138628 2024-04-25T04:33:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T04:33:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T04:33:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T04:33:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T04:33:33.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T04:33:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T04:33:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T04:33:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T04:33:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T04:33:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T04:33:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144611 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:33:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T04:33:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:33:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:33:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:33:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:33:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:33:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:33:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:33:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:33:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:33:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:33:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:33:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=d86f6f00f01208c31f9ea20303a857f2 2024-04-25T04:34:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T04:34:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T04:34:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:28.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T04:34:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=5ef60c6e859d08f7b69b59560a302696 2024-04-25T04:34:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T04:34:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T04:34:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:28.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T04:34:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T04:34:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:34:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:34:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:34:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=0b7533646e977f2aa1c73874ca852aab 2024-04-25T04:34:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T04:34:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T04:34:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T04:34:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=7fca34e428f038c7cd9d27a218e010eb 2024-04-25T04:34:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T04:34:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=fb2602cebd251bce000aa9fcbb373cd9 2024-04-25T04:34:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T04:34:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=94d83d9c727909a10680a660d5d6eb5e 2024-04-25T04:34:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T04:34:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T04:34:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:31.326 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T04:34:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T04:34:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=6f2b36f5087c81b7e384baef24c54f83 2024-04-25T04:34:31.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T04:34:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T04:34:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T04:34:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=855a9de6454c33655ef9fb582949d81d 2024-04-25T04:34:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T04:34:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T04:34:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T04:34:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T04:34:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:31.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=efd2d339c620e415f10c907c3d9b753f 2024-04-25T04:34:31.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:34:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:34:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:34:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T04:34:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T04:34:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:34:32.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:34:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=76219f819794e6f0c26e7c5b56bd7565 2024-04-25T04:34:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T04:34:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T04:34:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:32.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T04:34:32.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:34:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:34:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:32.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T04:34:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:32.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:34:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6f3801fd-9c66-4c6b-9c17-42356988b18b 2024-04-25T04:34:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:32.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:33.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T04:34:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T04:34:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T04:34:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:34:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:34:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:33.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:33.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:34:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:34:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4aa05b27-d3f3-4b02-9202-0be6094130ea 2024-04-25T04:34:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:34:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6818b672-2285-4be4-a1ae-7405485002e6 2024-04-25T04:34:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:34:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9538dbfc-4e8b-447c-a5fa-3968992771aa 2024-04-25T04:34:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:34:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:34:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:34:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:34:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:34:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:34.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:34:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:34:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:34:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:34:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3b4e86f7-dfe9-45c5-8e30-08fda287cc10 2024-04-25T04:34:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:34.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab15d283-8f3b-4ab3-8239-581e191f477d 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a5edbcf7-797e-403b-a818-2107011039af 2024-04-25T04:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:34:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:34:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6f3801fd-9c66-4c6b-9c17-42356988b18b 2024-04-25T04:34:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:35.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:34:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:34:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:34:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:34:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4aa05b27-d3f3-4b02-9202-0be6094130ea 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6818b672-2285-4be4-a1ae-7405485002e6 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:34:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9538dbfc-4e8b-447c-a5fa-3968992771aa 2024-04-25T04:34:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:34:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:36.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:34:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:34:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:36.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3b4e86f7-dfe9-45c5-8e30-08fda287cc10 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab15d283-8f3b-4ab3-8239-581e191f477d 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:34:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a5edbcf7-797e-403b-a818-2107011039af 2024-04-25T04:34:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:34:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:34:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: cd0e1482-d103-434e-b5f9-275b2b7ff625 2024-04-25T04:34:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:37.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:34:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:34:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:34:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:34:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6f3801fd-9c66-4c6b-9c17-42356988b18b 2024-04-25T04:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:34:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:39.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:34:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:34:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: cd0e1482-d103-434e-b5f9-275b2b7ff625 2024-04-25T04:34:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:34:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:34:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:34:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:34:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6818b672-2285-4be4-a1ae-7405485002e6 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4aa05b27-d3f3-4b02-9202-0be6094130ea 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:40.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:40.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:34:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9538dbfc-4e8b-447c-a5fa-3968992771aa 2024-04-25T04:34:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:41.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:34:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:34:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:34:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab15d283-8f3b-4ab3-8239-581e191f477d 2024-04-25T04:34:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:34:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3b4e86f7-dfe9-45c5-8e30-08fda287cc10 2024-04-25T04:34:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:34:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a5edbcf7-797e-403b-a818-2107011039af 2024-04-25T04:34:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:41.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:34:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: cd0e1482-d103-434e-b5f9-275b2b7ff625 2024-04-25T04:34:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:34:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:34:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:34:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:34:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:44.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:34:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:34:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:34:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:34:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:34:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:34:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:16' 2024-04-25T04:34:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:34:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:34:47.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:34:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6f3801fd-9c66-4c6b-9c17-42356988b18b 2024-04-25T04:34:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:47.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:47.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:34:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:34:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:34:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:34:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:34:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6818b672-2285-4be4-a1ae-7405485002e6 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4aa05b27-d3f3-4b02-9202-0be6094130ea 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:34:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9538dbfc-4e8b-447c-a5fa-3968992771aa 2024-04-25T04:34:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:48.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:34:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:34:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:34:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:34:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:34:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:34:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:34:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6f3801fd-9c66-4c6b-9c17-42356988b18b 2024-04-25T04:34:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:48.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:48.618 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:34:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:34:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T04:34:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:34:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T04:34:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T04:34:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:34:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:34:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3b4e86f7-dfe9-45c5-8e30-08fda287cc10 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab15d283-8f3b-4ab3-8239-581e191f477d 2024-04-25T04:34:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T04:34:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:34:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a5edbcf7-797e-403b-a818-2107011039af 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:49.234 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:34:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:34:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:34:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:34:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:34:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T04:34:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:34:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T04:34:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T04:34:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T04:34:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T04:34:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T04:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T04:34:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:34:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9538dbfc-4e8b-447c-a5fa-3968992771aa 2024-04-25T04:34:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6818b672-2285-4be4-a1ae-7405485002e6 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4aa05b27-d3f3-4b02-9202-0be6094130ea 2024-04-25T04:34:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:34:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:34:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:34:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:34:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:34:49.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:34:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T04:34:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T04:34:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T04:34:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T04:34:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T04:34:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T04:34:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:34:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T04:34:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:34:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T04:34:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:34:50.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T04:34:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:34:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab15d283-8f3b-4ab3-8239-581e191f477d 2024-04-25T04:34:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:34:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3b4e86f7-dfe9-45c5-8e30-08fda287cc10 2024-04-25T04:34:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:34:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a5edbcf7-797e-403b-a818-2107011039af 2024-04-25T04:34:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:34:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:50.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:34:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T04:34:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:34:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T04:34:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:34:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:34:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:34:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:34:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T04:34:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T04:34:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:50.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:34:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T04:34:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T04:34:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T04:34:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T04:34:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T04:34:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:34:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T04:34:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:34:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T04:34:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:34:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:34:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:52.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:34:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:34:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: cd0e1482-d103-434e-b5f9-275b2b7ff625 2024-04-25T04:34:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:34:52.489 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:34:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:52.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:34:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:34:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:34:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:34:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:52.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:34:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:34:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T04:34:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T04:34:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T04:34:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:34:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: cd0e1482-d103-434e-b5f9-275b2b7ff625 2024-04-25T04:34:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:53.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:34:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:34:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T04:34:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:34:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:54.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:34:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:34:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:34:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:54.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:34:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:34:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:34:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:34:54.610 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:34:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T04:34:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:34:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T04:34:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:34:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:34:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T04:34:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:34:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:34:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T04:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T04:34:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T04:34:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:34:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T04:34:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:34:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:34:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T04:34:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:34:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:34:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T04:34:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:34:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:34:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:34:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:34:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:34:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:34:46' 2024-04-25T04:34:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:34:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:34:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:34:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:34:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:34:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:34:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:34:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:34:55.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:34:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:34:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:55.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 04:34:46' 2024-04-25T04:34:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:34:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:34:46' 2024-04-25T04:34:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:34:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:34:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:34:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:34:46' 2024-04-25T04:34:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:34:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:34:56.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:34:46' 2024-04-25T04:34:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:34:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:34:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:34:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:34:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:34:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T04:34:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:34:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:34:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T04:34:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T04:34:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:34:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:35:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=0b7533646e977f2aa1c73874ca852aab 2024-04-25T04:35:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T04:35:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=94d83d9c727909a10680a660d5d6eb5e 2024-04-25T04:35:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T04:35:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 0b7533646e977f2aa1c73874ca852aab != \0\b\7\5\3\3\6\4\6\e\9\7\7\f\2\a\a\1\c\7\3\8\7\4\c\a\8\5\2\a\a\b ]] 2024-04-25T04:35:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 94d83d9c727909a10680a660d5d6eb5e != \9\4\d\8\3\d\9\c\7\2\7\9\0\9\a\1\0\6\8\0\a\6\6\0\d\5\d\6\e\b\5\e ]] 2024-04-25T04:35:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=7fca34e428f038c7cd9d27a218e010eb 2024-04-25T04:35:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T04:35:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 7fca34e428f038c7cd9d27a218e010eb != \7\f\c\a\3\4\e\4\2\8\f\0\3\8\c\7\c\d\9\d\2\7\a\2\1\8\e\0\1\0\e\b ]] 2024-04-25T04:35:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=fb2602cebd251bce000aa9fcbb373cd9 2024-04-25T04:35:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T04:35:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=6f2b36f5087c81b7e384baef24c54f83 2024-04-25T04:35:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T04:35:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ fb2602cebd251bce000aa9fcbb373cd9 != \f\b\2\6\0\2\c\e\b\d\2\5\1\b\c\e\0\0\0\a\a\9\f\c\b\b\3\7\3\c\d\9 ]] 2024-04-25T04:35:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=76219f819794e6f0c26e7c5b56bd7565 2024-04-25T04:35:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T04:35:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 6f2b36f5087c81b7e384baef24c54f83 != \6\f\2\b\3\6\f\5\0\8\7\c\8\1\b\7\e\3\8\4\b\a\e\f\2\4\c\5\4\f\8\3 ]] 2024-04-25T04:35:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 76219f819794e6f0c26e7c5b56bd7565 != \7\6\2\1\9\f\8\1\9\7\9\4\e\6\f\0\c\2\6\e\7\c\5\b\5\6\b\d\7\5\6\5 ]] 2024-04-25T04:35:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=855a9de6454c33655ef9fb582949d81d 2024-04-25T04:35:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T04:35:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 855a9de6454c33655ef9fb582949d81d != \8\5\5\a\9\d\e\6\4\5\4\c\3\3\6\5\5\e\f\9\f\b\5\8\2\9\4\9\d\8\1\d ]] 2024-04-25T04:35:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=efd2d339c620e415f10c907c3d9b753f 2024-04-25T04:35:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T04:35:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ efd2d339c620e415f10c907c3d9b753f != \e\f\d\2\d\3\3\9\c\6\2\0\e\4\1\5\f\1\0\c\9\0\7\c\3\d\9\b\7\5\3\f ]] 2024-04-25T04:35:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144612 2024-04-25T04:35:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=5ef60c6e859d08f7b69b59560a302696 2024-04-25T04:35:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T04:35:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=d86f6f00f01208c31f9ea20303a857f2 2024-04-25T04:35:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T04:35:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 5ef60c6e859d08f7b69b59560a302696 != \5\e\f\6\0\c\6\e\8\5\9\d\0\8\f\7\b\6\9\b\5\9\5\6\0\a\3\0\2\6\9\6 ]] 2024-04-25T04:35:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144614 2024-04-25T04:35:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144616 2024-04-25T04:35:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144618 2024-04-25T04:35:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144620 2024-04-25T04:35:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144622 2024-04-25T04:35:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ d86f6f00f01208c31f9ea20303a857f2 != \d\8\6\f\6\f\0\0\f\0\1\2\0\8\c\3\1\f\9\e\a\2\0\3\0\3\a\8\5\7\f\2 ]] 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144624 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144626 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 144628 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:51.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:35:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:35:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T04:35:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:35:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:35:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:35:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:35:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: cd0e1482-d103-434e-b5f9-275b2b7ff625 2024-04-25T04:35:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:35:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3689472.0,"bytes_per_snapshot":73789440.0,"last_snapshot_bytes":73789440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"} 2024-04-25T04:35:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:35:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:35:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:35:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:35:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:35:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3689472.0,"bytes_per_snapshot":73789440.0,"last_snapshot_bytes":73789440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"} 2024-04-25T04:35:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:35:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:35:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:35:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:35:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:35:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:35:52.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:35:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:35:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:35:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:35:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:35:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:35:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:35:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:35:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1cd0e1482-d103-434e-b5f9-275b2b7ff625up+replayingreplaying, {"bytes_per_second":3689472.0,"bytes_per_snapshot":73789440.0,"last_snapshot_bytes":73789440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+replayingreplaying, {"bytes_per_second":3371417.6,"bytes_per_snapshot":67428352.0,"last_snapshot_bytes":67428352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019697,"remote_snapshot_timestamp":1714019697,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+replayingreplaying, {"bytes_per_second":3307110.4,"bytes_per_snapshot":66142208.0,"last_snapshot_bytes":66142208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+replayingreplaying, {"bytes_per_second":3610009.6,"bytes_per_snapshot":72200192.0,"last_snapshot_bytes":72200192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+replayingreplaying, {"bytes_per_second":3606323.2,"bytes_per_snapshot":72126464.0,"last_snapshot_bytes":72126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+replayingreplaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+replayingreplaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:35:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:35:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:35:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:35:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:35:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:35:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:35:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:35:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:35:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T04:35:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:35:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:35:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T04:35:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T04:35:53.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:35:53.058+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c62ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:35:53.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:35:53.058+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb680 [3/cd0e1482-d103-434e-b5f9-275b2b7ff625] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:35:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:35:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T04:35:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:35:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:35:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:35:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T04:35:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T04:35:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:35:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:35:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:35:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:35:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:35:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T04:35:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:35:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:35:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:35:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:35:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:35:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:35:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:35:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 2d407fb8-b623-40d2-a78a-30fff62abfad 2024-04-25T04:35:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:35:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3371417.6,"bytes_per_snapshot":67428352.0,"last_snapshot_bytes":67428352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019697,"remote_snapshot_timestamp":1714019697,"replay_state":"idle"} 2024-04-25T04:35:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:35:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:35:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:35:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:35:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:35:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3371417.6,"bytes_per_snapshot":67428352.0,"last_snapshot_bytes":67428352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019697,"remote_snapshot_timestamp":1714019697,"replay_state":"idle"} 2024-04-25T04:35:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:35:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:35:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:35:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:35:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:35:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:35:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:35:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:35:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:35:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:35:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:35:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:35:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:35:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:35:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:35:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary22d407fb8-b623-40d2-a78a-30fff62abfadup+replayingreplaying, {"bytes_per_second":3371417.6,"bytes_per_snapshot":67428352.0,"last_snapshot_bytes":67428352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019697,"remote_snapshot_timestamp":1714019697,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+replayingreplaying, {"bytes_per_second":3307110.4,"bytes_per_snapshot":66142208.0,"last_snapshot_bytes":66142208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+replayingreplaying, {"bytes_per_second":3610009.6,"bytes_per_snapshot":72200192.0,"last_snapshot_bytes":72200192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+replayingreplaying, {"bytes_per_second":3606323.2,"bytes_per_snapshot":72126464.0,"last_snapshot_bytes":72126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+replayingreplaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+replayingreplaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:35:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:35:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:35:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:35:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:35:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:35:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:35:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:35:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:35:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T04:35:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:35:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:35:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T04:35:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T04:35:55.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:35:55.805+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664480680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:35:55.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:35:55.805+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/2d407fb8-b623-40d2-a78a-30fff62abfad] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:35:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:35:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T04:35:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:35:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:35:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:35:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T04:35:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T04:35:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:35:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:35:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T04:35:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:35:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:35:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:35:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T04:35:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:35:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:35:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:35:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:35:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:35:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:35:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:35:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ab15d283-8f3b-4ab3-8239-581e191f477d 2024-04-25T04:35:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:35:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3307110.4,"bytes_per_snapshot":66142208.0,"last_snapshot_bytes":66142208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"} 2024-04-25T04:35:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:35:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:35:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:35:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:35:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:35:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3307110.4,"bytes_per_snapshot":66142208.0,"last_snapshot_bytes":66142208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"} 2024-04-25T04:35:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:35:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:35:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:35:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:35:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:35:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:35:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:35:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:35:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:35:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:35:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:35:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary3ab15d283-8f3b-4ab3-8239-581e191f477dup+replayingreplaying, {"bytes_per_second":3307110.4,"bytes_per_snapshot":66142208.0,"last_snapshot_bytes":66142208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+replayingreplaying, {"bytes_per_second":3610009.6,"bytes_per_snapshot":72200192.0,"last_snapshot_bytes":72200192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+replayingreplaying, {"bytes_per_second":3606323.2,"bytes_per_snapshot":72126464.0,"last_snapshot_bytes":72126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+replayingreplaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+replayingreplaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:35:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:35:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:35:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:35:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:35:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:35:58.614 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:35:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:35:58.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:35:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:35:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T04:35:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:35:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:35:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:35:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T04:35:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T04:35:58.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:35:58.824+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966810c680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:35:58.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:35:58.824+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508a00 [3/ab15d283-8f3b-4ab3-8239-581e191f477d] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T04:36:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:00.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T04:36:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T04:36:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:36:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T04:36:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:01.416 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:36:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:36:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 3b4e86f7-dfe9-45c5-8e30-08fda287cc10 2024-04-25T04:36:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:36:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3610009.6,"bytes_per_snapshot":72200192.0,"last_snapshot_bytes":72200192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"} 2024-04-25T04:36:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:36:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:36:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:36:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:36:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:36:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3610009.6,"bytes_per_snapshot":72200192.0,"last_snapshot_bytes":72200192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"} 2024-04-25T04:36:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:36:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:36:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:36:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:36:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:36:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:36:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary43b4e86f7-dfe9-45c5-8e30-08fda287cc10up+replayingreplaying, {"bytes_per_second":3610009.6,"bytes_per_snapshot":72200192.0,"last_snapshot_bytes":72200192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+replayingreplaying, {"bytes_per_second":3606323.2,"bytes_per_snapshot":72126464.0,"last_snapshot_bytes":72126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+replayingreplaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+replayingreplaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:36:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:36:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:36:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:36:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:36:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:36:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T04:36:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:36:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T04:36:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T04:36:01.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:01.985+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664f0b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:36:01.989 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:01.985+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/3b4e86f7-dfe9-45c5-8e30-08fda287cc10] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T04:36:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T04:36:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T04:36:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:02.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:02.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:36:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:36:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4aa05b27-d3f3-4b02-9202-0be6094130ea 2024-04-25T04:36:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:36:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3606323.2,"bytes_per_snapshot":72126464.0,"last_snapshot_bytes":72126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"} 2024-04-25T04:36:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:36:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:36:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:36:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:36:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:36:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3606323.2,"bytes_per_snapshot":72126464.0,"last_snapshot_bytes":72126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"} 2024-04-25T04:36:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:36:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:36:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:36:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:36:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:36:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:36:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:36:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:36:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:36:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary54aa05b27-d3f3-4b02-9202-0be6094130eaup+replayingreplaying, {"bytes_per_second":3606323.2,"bytes_per_snapshot":72126464.0,"last_snapshot_bytes":72126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+replayingreplaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+replayingreplaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:36:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:36:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:36:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:36:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:36:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:36:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T04:36:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:36:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T04:36:04.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T04:36:04.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:04.251+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667879520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:36:04.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:04.251+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda780 [3/4aa05b27-d3f3-4b02-9202-0be6094130ea] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T04:36:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:06.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T04:36:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T04:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6f3801fd-9c66-4c6b-9c17-42356988b18b 2024-04-25T04:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"} 2024-04-25T04:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:36:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:36:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:36:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:36:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"} 2024-04-25T04:36:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:36:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:36:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:36:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:36:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:36:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:36:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:36:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:36:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:36:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary66f3801fd-9c66-4c6b-9c17-42356988b18bup+replayingreplaying, {"bytes_per_second":3138764.8,"bytes_per_snapshot":62775296.0,"last_snapshot_bytes":62775296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019688,"remote_snapshot_timestamp":1714019688,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+replayingreplaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:36:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:36:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:36:07.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:36:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:36:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:36:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T04:36:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:36:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T04:36:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T04:36:07.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:07.913+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596684824e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:36:07.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:07.913+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x5596655f6000 [3/6f3801fd-9c66-4c6b-9c17-42356988b18b] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T04:36:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:09.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T04:36:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T04:36:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:36:09.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:10.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:36:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:36:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ac34346b-9fa2-404c-a7b0-3de751d356f4 2024-04-25T04:36:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:36:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"} 2024-04-25T04:36:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:36:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:36:10.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:36:10.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:36:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:36:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"} 2024-04-25T04:36:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:36:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:36:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:36:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:36:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:36:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:36:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary7ac34346b-9fa2-404c-a7b0-3de751d356f4up+replayingreplaying, {"bytes_per_second":3274342.4,"bytes_per_snapshot":65486848.0,"last_snapshot_bytes":65486848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019695,"remote_snapshot_timestamp":1714019695,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:36:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:36:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:36:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:36:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:36:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:36:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T04:36:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:36:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:11.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T04:36:11.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T04:36:11.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:11.281+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966567e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:36:11.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:11.281+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/ac34346b-9fa2-404c-a7b0-3de751d356f4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T04:36:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T04:36:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:36:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T04:36:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:36:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:36:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: a5edbcf7-797e-403b-a818-2107011039af 2024-04-25T04:36:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:36:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"} 2024-04-25T04:36:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:36:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:35:46' 2024-04-25T04:36:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:36:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:36:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:36:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"} 2024-04-25T04:36:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:36:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:36:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:36:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:36:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:36:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 04:35:46' 2024-04-25T04:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:35:46' 2024-04-25T04:36:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:36:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":3388006.4,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary8a5edbcf7-797e-403b-a818-2107011039afup+replayingreplaying, {"bytes_per_second":3673702.4,"bytes_per_snapshot":73474048.0,"last_snapshot_bytes":73474048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019691,"remote_snapshot_timestamp":1714019691,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":3227443.2,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:35:46' 2024-04-25T04:36:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:36:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:36:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:35:46' 2024-04-25T04:36:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:36:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:36:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:36:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T04:36:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:36:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T04:36:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T04:36:14.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:14.255+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664f0a000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:36:14.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:14.255+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508780 [3/a5edbcf7-797e-403b-a818-2107011039af] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T04:36:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T04:36:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T04:36:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:36:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:16.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:36:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:36:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6818b672-2285-4be4-a1ae-7405485002e6 2024-04-25T04:36:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:36:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"} 2024-04-25T04:36:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:36:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:36:16' 2024-04-25T04:36:16.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:36:16.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:36:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"} 2024-04-25T04:36:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:36:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:36:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:36:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:36:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:36:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:36:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:36:16' 2024-04-25T04:36:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 04:36:16' 2024-04-25T04:36:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:36:16' 2024-04-25T04:36:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:36:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:36:16image-primary96818b672-2285-4be4-a1ae-7405485002e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64548864.0,"last_snapshot_bytes":64548864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:36:16' 2024-04-25T04:36:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:36:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:36:17.035 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:36:16' 2024-04-25T04:36:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:36:17.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:36:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:36:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T04:36:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:36:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T04:36:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T04:36:17.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:17.256+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596685d7520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:36:17.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:17.256+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdb400 [3/6818b672-2285-4be4-a1ae-7405485002e6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T04:36:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:18.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T04:36:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T04:36:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:36:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T04:36:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:36:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T04:36:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:36:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:36:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:36:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:36:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 9538dbfc-4e8b-447c-a5fa-3968992771aa 2024-04-25T04:36:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:36:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"} 2024-04-25T04:36:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:36:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:36:16' 2024-04-25T04:36:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:36:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:36:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:36:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"} 2024-04-25T04:36:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:36:19.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:36:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:36:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:36:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:36:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:36:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:36:16' 2024-04-25T04:36:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 04:36:16' 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:36:16' 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:36:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:36:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary109538dbfc-4e8b-447c-a5fa-3968992771aaup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67760128.0,"last_snapshot_bytes":67760128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714019690,"remote_snapshot_timestamp":1714019690,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:36:16' 2024-04-25T04:36:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:36:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:36:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:36:16' 2024-04-25T04:36:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:36:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:36:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:36:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:36:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:36:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:20.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T04:36:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:36:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T04:36:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T04:36:20.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:20.252+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c5ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:36:20.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:20.252+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508000 [3/9538dbfc-4e8b-447c-a5fa-3968992771aa] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:36:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:36:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T04:36:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:36:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:36:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T04:36:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T04:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..10} 2024-04-25T04:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS=() 2024-04-25T04:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS=() 2024-04-25T04:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS=() 2024-04-25T04:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS=() 2024-04-25T04:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ RET=0 2024-04-25T04:36:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary1 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-04-25T04:36:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-04-25T04:36:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary1': 2024-04-25T04:36:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3ce52fa8d9c 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3ce52fa8d9c 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:22 2024 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:22 2024 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:22 2024 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 6c3ebfcc-0796-4a1c-8804-a70579d0f02e 2024-04-25T04:36:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-04-25T04:36:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd0 2024-04-25T04:36:22.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-04-25T04:36:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/26214401572864/2621440 done 2024-04-25T04:36:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: f8b0752b-5d7a-4fd7-a360-747d0358327b 2024-04-25T04:36:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:23.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary1 2024-04-25T04:36:23.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-04-25T04:36:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary1 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary2 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary1 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-04-25T04:36:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-04-25T04:36:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2974 2024-04-25T04:36:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary2': 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3cf5143702f3 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3cf5143702f3 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:23 2024 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:23 2024 2024-04-25T04:36:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:23 2024 2024-04-25T04:36:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 8e3fcaef-3fd6-4c33-9da0-5f6e4433df39 2024-04-25T04:36:23.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-04-25T04:36:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd1 2024-04-25T04:36:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-04-25T04:36:23.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: fe04ae18-b4e8-4c0b-b86a-41a311bccb26 2024-04-25T04:36:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary2 2024-04-25T04:36:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-04-25T04:36:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary2 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary3 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary2 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary2 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:24.142 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-04-25T04:36:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-04-25T04:36:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2976 2024-04-25T04:36:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-04-25T04:36:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary3': 2024-04-25T04:36:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d0479bfc50c 2024-04-25T04:36:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d0479bfc50c 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:24 2024 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:24 2024 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:24 2024 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: f89850dc-5bb7-460e-9d8e-c7d8d0b23af6 2024-04-25T04:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-04-25T04:36:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd2 2024-04-25T04:36:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-04-25T04:36:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 9e93ccc3-b7ba-4f02-8606-c1992f9a01c1 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary3 2024-04-25T04:36:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-04-25T04:36:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary3 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:25.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary3 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary4 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary3 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-04-25T04:36:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-04-25T04:36:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2978 2024-04-25T04:36:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-04-25T04:36:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary4': 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d135490d1b3 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d135490d1b3 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:25 2024 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:25 2024 2024-04-25T04:36:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:25 2024 2024-04-25T04:36:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 4a0ff156-3598-4982-991b-a69c3615b291 2024-04-25T04:36:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-04-25T04:36:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd3 2024-04-25T04:36:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-04-25T04:36:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ca5ca880-a971-4a08-8c0a-aaf5efaf180c 2024-04-25T04:36:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary4 2024-04-25T04:36:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-04-25T04:36:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary4 2024-04-25T04:36:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:36:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary4 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary5 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary4 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-04-25T04:36:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:26.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-04-25T04:36:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2980 2024-04-25T04:36:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-04-25T04:36:26.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary5': 2024-04-25T04:36:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d22ccedf002 2024-04-25T04:36:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d22ccedf002 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:26 2024 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:26 2024 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:26 2024 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:36:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-04-25T04:36:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd4 2024-04-25T04:36:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-04-25T04:36:27.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 451b2059-045d-42dc-a544-96e92318d69b 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary5 2024-04-25T04:36:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-04-25T04:36:27.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary5 2024-04-25T04:36:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary5 2024-04-25T04:36:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:36:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary6 2024-04-25T04:36:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-04-25T04:36:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary5 2024-04-25T04:36:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-04-25T04:36:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:27.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2983 2024-04-25T04:36:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2982 2024-04-25T04:36:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-04-25T04:36:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2985 2024-04-25T04:36:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary6': 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d34ec3afd97 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d34ec3afd97 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:27 2024 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:27 2024 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:27 2024 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 0e2cdce8-7612-40bd-84f8-e6b644244c10 2024-04-25T04:36:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-04-25T04:36:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd5 2024-04-25T04:36:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-04-25T04:36:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: ab9273c4-77e3-4c0f-b5a3-6749129b9fd6 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary6 2024-04-25T04:36:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-04-25T04:36:29.285 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary6 2024-04-25T04:36:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:36:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary7 2024-04-25T04:36:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary6 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary6 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-04-25T04:36:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-04-25T04:36:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2987 2024-04-25T04:36:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2986 2024-04-25T04:36:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-04-25T04:36:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary7': 2024-04-25T04:36:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d48f712911e 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d48f712911e 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:29 2024 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:29 2024 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:29 2024 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ff549457-25a5-4b41-922c-5c7a9dcb33a4 2024-04-25T04:36:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-04-25T04:36:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd6 2024-04-25T04:36:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-04-25T04:36:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 086d6ac5-9bce-46a0-a1ec-a1406525b9d9 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary7 2024-04-25T04:36:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-04-25T04:36:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:30.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary7 2024-04-25T04:36:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-04-25T04:36:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary8 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary7 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary7 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-04-25T04:36:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-04-25T04:36:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-04-25T04:36:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-04-25T04:36:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2990 2024-04-25T04:36:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2989 2024-04-25T04:36:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-04-25T04:36:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary8': 2024-04-25T04:36:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d59361805b 2024-04-25T04:36:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d59361805b 2024-04-25T04:36:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:30 2024 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:30 2024 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:30 2024 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: ec2399e4-6cb9-4060-91b4-d0aab89b8fa7 2024-04-25T04:36:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-04-25T04:36:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd7 2024-04-25T04:36:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-04-25T04:36:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 215924fe-1d7b-4430-a3de-b3a629708428 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary8 2024-04-25T04:36:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-04-25T04:36:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary8 2024-04-25T04:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-04-25T04:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary8 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary9 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary8 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-04-25T04:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-04-25T04:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-04-25T04:36:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-04-25T04:36:31.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2994 2024-04-25T04:36:31.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2992 2024-04-25T04:36:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2993 2024-04-25T04:36:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:31.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:31.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:31.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-04-25T04:36:31.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary9': 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d6c802390ce 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d6c802390ce 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:31 2024 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:31 2024 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:31 2024 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 1a6cb9ad-5b91-4698-af22-d148427b9c03 2024-04-25T04:36:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-04-25T04:36:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd8 2024-04-25T04:36:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-04-25T04:36:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/26214401048576/2621440 done 2024-04-25T04:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: e74f398e-ca94-4686-8894-4205e341155a 2024-04-25T04:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary9 2024-04-25T04:36:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-04-25T04:36:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary9 2024-04-25T04:36:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary9 2024-04-25T04:36:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ MNTPT=test-primary10 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary9 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n snapshot ']' 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local size=128 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n 10G ']' 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ size=10G 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ shift 2024-04-25T04:36:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-04-25T04:36:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ is_pool_mirror_mode_image mirror 2024-04-25T04:36:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' image = image ']' 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mode=snapshot 2024-04-25T04:36:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-04-25T04:36:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-04-25T04:36:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2997 2024-04-25T04:36:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Mirroring enabled 2024-04-25T04:36:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2996 2024-04-25T04:36:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:32.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-04-25T04:36:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:rbd image 'image-primary10': 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: size 10 GiB in 2560 objects 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: order 22 (4 MiB objects) 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: snapshot_count: 1 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: id: 3d7c684433fc 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: block_name_prefix: rbd_data.3d7c684433fc 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: format: 2 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-04-25T04:36:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: op_features: 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: flags: 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: create_timestamp: Thu Apr 25 04:36:32 2024 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: access_timestamp: Thu Apr 25 04:36:32 2024 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: modify_timestamp: Thu Apr 25 04:36:32 2024 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring state: enabled 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring mode: snapshot 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring global id: 35ac40cd-f116-41a1-9df2-1315e256f174 2024-04-25T04:36:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: mirroring primary: true 2024-04-25T04:36:32.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:36:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:36:32.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-04-25T04:36:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEV=/dev/rbd9 2024-04-25T04:36:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ DEVS+=($DEV) 2024-04-25T04:36:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-04-25T04:36:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:mke2fs 1.45.6 (20-Mar-2020) 2024-04-25T04:36:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Discarding device blocks: 0/2621440 done 2024-04-25T04:36:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-04-25T04:36:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Filesystem UUID: 90032727-4f65-48f6-b03a-d5a8681faa17 2024-04-25T04:36:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Superblock backups stored on blocks: 2024-04-25T04:36:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-04-25T04:36:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Allocating group tables: 0/80 done 2024-04-25T04:36:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing inode tables: 0/80 done 2024-04-25T04:36:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Creating journal (16384 blocks): done 2024-04-25T04:36:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-04-25T04:36:33.129 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: 2024-04-25T04:36:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mkdir test-primary10 2024-04-25T04:36:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-04-25T04:36:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ whoami 2024-04-25T04:36:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo chown ubuntu test-primary10 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ SNAP_PIDS+=($!) 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ WORKLOAD_PIDS+=($!) 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ slow_untar_workload test-primary10 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 148980 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mountpt=test-primary10 2024-04-25T04:36:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-04-25T04:36:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local ret=0 2024-04-25T04:36:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-04-25T04:36:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3000 2024-04-25T04:36:33.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 2999 2024-04-25T04:36:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3001 2024-04-25T04:36:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:36:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:36:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:34.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:34.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3002 2024-04-25T04:36:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3003 2024-04-25T04:36:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3006 2024-04-25T04:36:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3005 2024-04-25T04:36:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:36:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:36:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3004 2024-04-25T04:36:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:35.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:36.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:36:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:36:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3008 2024-04-25T04:36:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3007 2024-04-25T04:36:36.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:37.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3010 2024-04-25T04:36:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3011 2024-04-25T04:36:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3009 2024-04-25T04:36:38.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3012 2024-04-25T04:36:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3013 2024-04-25T04:36:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:38.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:38.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:36:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:36:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3016 2024-04-25T04:36:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3014 2024-04-25T04:36:39.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:36:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:36:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3015 2024-04-25T04:36:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:40.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3017 2024-04-25T04:36:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3018 2024-04-25T04:36:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:36:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:36:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:41.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3020 2024-04-25T04:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3021 2024-04-25T04:36:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3019 2024-04-25T04:36:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:41.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:42.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3023 2024-04-25T04:36:42.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3022 2024-04-25T04:36:42.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:36:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:36:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3024 2024-04-25T04:36:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3025 2024-04-25T04:36:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:43.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:36:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:36:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3026 2024-04-25T04:36:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3028 2024-04-25T04:36:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:36:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:36:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:44.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:45.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3029 2024-04-25T04:36:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3027 2024-04-25T04:36:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3031 2024-04-25T04:36:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:45.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3030 2024-04-25T04:36:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:36:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:46.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:36:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3033 2024-04-25T04:36:46.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:36:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:36:47.767 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:36:47.764+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657de000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:36:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3035 2024-04-25T04:36:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3032 2024-04-25T04:36:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3034 2024-04-25T04:36:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:36:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3036 2024-04-25T04:36:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3037 2024-04-25T04:36:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:49.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3039 2024-04-25T04:36:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3038 2024-04-25T04:36:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3040 2024-04-25T04:36:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:36:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:36:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3041 2024-04-25T04:36:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3042 2024-04-25T04:36:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:36:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:36:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3044 2024-04-25T04:36:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:36:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3043 2024-04-25T04:36:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3046 2024-04-25T04:36:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3045 2024-04-25T04:36:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:53.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3048 2024-04-25T04:36:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3047 2024-04-25T04:36:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3050 2024-04-25T04:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:55.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:36:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3049 2024-04-25T04:36:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:36:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:36:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:36:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:36:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:36:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:36:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3052 2024-04-25T04:36:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:36:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:36:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:36:56.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3051 2024-04-25T04:36:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3053 2024-04-25T04:36:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:36:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:36:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:36:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:36:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:36:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:36:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3055 2024-04-25T04:36:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3054 2024-04-25T04:36:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3056 2024-04-25T04:36:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:36:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:36:58.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:36:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:36:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:36:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:36:59.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3058 2024-04-25T04:36:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:36:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:36:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:36:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:36:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:36:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:36:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:36:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:36:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:36:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:36:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:36:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:36:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3057 2024-04-25T04:37:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3059 2024-04-25T04:37:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3063 2024-04-25T04:37:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:00.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:00.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3060 2024-04-25T04:37:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3061 2024-04-25T04:37:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3062 2024-04-25T04:37:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3064 2024-04-25T04:37:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3065 2024-04-25T04:37:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:03.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3066 2024-04-25T04:37:03.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:03.887 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3067 2024-04-25T04:37:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:04.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3070 2024-04-25T04:37:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3068 2024-04-25T04:37:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3071 2024-04-25T04:37:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:06.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3069 2024-04-25T04:37:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3072 2024-04-25T04:37:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:06.851 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3073 2024-04-25T04:37:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:07.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3074 2024-04-25T04:37:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3076 2024-04-25T04:37:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3075 2024-04-25T04:37:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:08.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3077 2024-04-25T04:37:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3078 2024-04-25T04:37:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:09.848 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:09.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3079 2024-04-25T04:37:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3081 2024-04-25T04:37:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:11.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:11.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:11.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3080 2024-04-25T04:37:11.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:12.669 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3083 2024-04-25T04:37:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3082 2024-04-25T04:37:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3084 2024-04-25T04:37:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:13.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3086 2024-04-25T04:37:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:13.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:37:13.787+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596657f2400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3087 2024-04-25T04:37:14.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3085 2024-04-25T04:37:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3088 2024-04-25T04:37:14.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3089 2024-04-25T04:37:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3090 2024-04-25T04:37:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3091 2024-04-25T04:37:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3093 2024-04-25T04:37:16.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3092 2024-04-25T04:37:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3094 2024-04-25T04:37:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3097 2024-04-25T04:37:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:18.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3095 2024-04-25T04:37:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3096 2024-04-25T04:37:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3098 2024-04-25T04:37:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3099 2024-04-25T04:37:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:20.821 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:37:20.819+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5596675dd000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:37:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:21.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3101 2024-04-25T04:37:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3100 2024-04-25T04:37:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3102 2024-04-25T04:37:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3103 2024-04-25T04:37:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:22.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3104 2024-04-25T04:37:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:23.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3106 2024-04-25T04:37:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3105 2024-04-25T04:37:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:24.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3109 2024-04-25T04:37:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3108 2024-04-25T04:37:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3111 2024-04-25T04:37:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3110 2024-04-25T04:37:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3107 2024-04-25T04:37:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3112 2024-04-25T04:37:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3113 2024-04-25T04:37:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3114 2024-04-25T04:37:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:27.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:27.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3115 2024-04-25T04:37:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3116 2024-04-25T04:37:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:29.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3119 2024-04-25T04:37:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3117 2024-04-25T04:37:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3118 2024-04-25T04:37:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:30.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:31.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3120 2024-04-25T04:37:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3122 2024-04-25T04:37:32.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3123 2024-04-25T04:37:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:32.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3124 2024-04-25T04:37:33.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3121 2024-04-25T04:37:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:33.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:33.837 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:37:33.835+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966b162000 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:37:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3127 2024-04-25T04:37:33.909 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3125 2024-04-25T04:37:34.763 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3126 2024-04-25T04:37:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:34.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3128 2024-04-25T04:37:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3129 2024-04-25T04:37:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3130 2024-04-25T04:37:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:35.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3132 2024-04-25T04:37:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3131 2024-04-25T04:37:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3133 2024-04-25T04:37:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3134 2024-04-25T04:37:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3135 2024-04-25T04:37:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3136 2024-04-25T04:37:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3137 2024-04-25T04:37:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3139 2024-04-25T04:37:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3138 2024-04-25T04:37:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3140 2024-04-25T04:37:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:40.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:40.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:41.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3141 2024-04-25T04:37:41.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3145 2024-04-25T04:37:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3142 2024-04-25T04:37:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3146 2024-04-25T04:37:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:43.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:43.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3144 2024-04-25T04:37:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3143 2024-04-25T04:37:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3147 2024-04-25T04:37:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3150 2024-04-25T04:37:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3148 2024-04-25T04:37:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:45.824 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:37:45.821+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55966573c400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:37:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:45.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:45.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3149 2024-04-25T04:37:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3151 2024-04-25T04:37:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:46.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3154 2024-04-25T04:37:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3153 2024-04-25T04:37:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3152 2024-04-25T04:37:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3156 2024-04-25T04:37:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3155 2024-04-25T04:37:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3157 2024-04-25T04:37:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3158 2024-04-25T04:37:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3159 2024-04-25T04:37:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3160 2024-04-25T04:37:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3163 2024-04-25T04:37:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3161 2024-04-25T04:37:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3162 2024-04-25T04:37:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3165 2024-04-25T04:37:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3166 2024-04-25T04:37:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3167 2024-04-25T04:37:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3168 2024-04-25T04:37:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3164 2024-04-25T04:37:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3169 2024-04-25T04:37:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3170 2024-04-25T04:37:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:37:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:37:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:37:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3172 2024-04-25T04:37:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3171 2024-04-25T04:37:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:37:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:37:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:37:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:37:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:37:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:37:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3175 2024-04-25T04:37:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3174 2024-04-25T04:37:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:37:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:37:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:37:58.675 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3173 2024-04-25T04:37:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3176 2024-04-25T04:37:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:37:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:37:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:37:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:37:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:37:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:37:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:37:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:37:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:37:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:37:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:37:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:37:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3179 2024-04-25T04:37:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3178 2024-04-25T04:37:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:37:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:37:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:37:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:37:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:37:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:37:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:37:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:37:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:37:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:37:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3180 2024-04-25T04:38:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3177 2024-04-25T04:38:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3181 2024-04-25T04:38:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3182 2024-04-25T04:38:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3184 2024-04-25T04:38:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3183 2024-04-25T04:38:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3186 2024-04-25T04:38:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3185 2024-04-25T04:38:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3187 2024-04-25T04:38:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3188 2024-04-25T04:38:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3189 2024-04-25T04:38:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3190 2024-04-25T04:38:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:06.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3192 2024-04-25T04:38:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3191 2024-04-25T04:38:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3193 2024-04-25T04:38:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3194 2024-04-25T04:38:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3195 2024-04-25T04:38:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3196 2024-04-25T04:38:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:08.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3198 2024-04-25T04:38:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3197 2024-04-25T04:38:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3200 2024-04-25T04:38:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3199 2024-04-25T04:38:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3201 2024-04-25T04:38:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3202 2024-04-25T04:38:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:11.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:11.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:12.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3204 2024-04-25T04:38:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3203 2024-04-25T04:38:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3206 2024-04-25T04:38:13.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3205 2024-04-25T04:38:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:13.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3207 2024-04-25T04:38:14.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3208 2024-04-25T04:38:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:14.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:14.758 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3210 2024-04-25T04:38:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3209 2024-04-25T04:38:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:15.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:16.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3211 2024-04-25T04:38:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3212 2024-04-25T04:38:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:16.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3213 2024-04-25T04:38:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3214 2024-04-25T04:38:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:17.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:17.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3215 2024-04-25T04:38:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3216 2024-04-25T04:38:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3218 2024-04-25T04:38:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3217 2024-04-25T04:38:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3219 2024-04-25T04:38:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3220 2024-04-25T04:38:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:20.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3221 2024-04-25T04:38:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3222 2024-04-25T04:38:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:22.755 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:38:22.753+0000 7fb929c00700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559667531400 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:38:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3223 2024-04-25T04:38:23.664 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3224 2024-04-25T04:38:23.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3225 2024-04-25T04:38:23.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3226 2024-04-25T04:38:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:23.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:23.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3228 2024-04-25T04:38:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3227 2024-04-25T04:38:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3229 2024-04-25T04:38:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3230 2024-04-25T04:38:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3232 2024-04-25T04:38:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3231 2024-04-25T04:38:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3234 2024-04-25T04:38:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3233 2024-04-25T04:38:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3235 2024-04-25T04:38:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3236 2024-04-25T04:38:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:28.809 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:29.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3238 2024-04-25T04:38:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3237 2024-04-25T04:38:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:29.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:29.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3240 2024-04-25T04:38:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3239 2024-04-25T04:38:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:30.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:30.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:31.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3241 2024-04-25T04:38:31.794 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3242 2024-04-25T04:38:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:31.810 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3244 2024-04-25T04:38:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:31.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:31.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:31.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:32.822 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:32.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:32.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3243 2024-04-25T04:38:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3245 2024-04-25T04:38:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3246 2024-04-25T04:38:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3247 2024-04-25T04:38:35.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3248 2024-04-25T04:38:35.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3251 2024-04-25T04:38:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3249 2024-04-25T04:38:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3250 2024-04-25T04:38:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3253 2024-04-25T04:38:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3252 2024-04-25T04:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:37.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3254 2024-04-25T04:38:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3256 2024-04-25T04:38:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3255 2024-04-25T04:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:39.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3259 2024-04-25T04:38:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3257 2024-04-25T04:38:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3258 2024-04-25T04:38:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3261 2024-04-25T04:38:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3260 2024-04-25T04:38:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:40.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3263 2024-04-25T04:38:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3262 2024-04-25T04:38:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:42.815 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-04-25T04:38:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:38:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-04-25T04:38:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3264 2024-04-25T04:38:43.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3265 2024-04-25T04:38:43.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3266 2024-04-25T04:38:43.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-04-25T04:38:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:38:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-04-25T04:38:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3268 2024-04-25T04:38:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3269 2024-04-25T04:38:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3267 2024-04-25T04:38:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-04-25T04:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:44.837 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:38:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-04-25T04:38:44.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:44.843 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3270 2024-04-25T04:38:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3271 2024-04-25T04:38:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3273 2024-04-25T04:38:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3272 2024-04-25T04:38:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-04-25T04:38:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:38:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-04-25T04:38:46.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-04-25T04:38:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:38:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-04-25T04:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3274 2024-04-25T04:38:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-04-25T04:38:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:38:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-04-25T04:38:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149114 2024-04-25T04:38:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-04-25T04:38:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:38:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-04-25T04:38:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149248 2024-04-25T04:38:48.772 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:38:48.769+0000 7fb92a600700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x559665747c00 handle_unlink_peer: failed to unlink local peer from remote image: (22) Invalid argument 2024-04-25T04:38:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3276 2024-04-25T04:38:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3275 2024-04-25T04:38:49.746 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-04-25T04:38:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:38:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-04-25T04:38:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149383 2024-04-25T04:38:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3277 2024-04-25T04:38:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3278 2024-04-25T04:38:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:50.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149538 2024-04-25T04:38:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3279 2024-04-25T04:38:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3280 2024-04-25T04:38:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-04-25T04:38:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:38:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-04-25T04:38:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149714 2024-04-25T04:38:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3281 2024-04-25T04:38:53.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for i in {1..30} 2024-04-25T04:38:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-04-25T04:38:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:38:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:38:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:38:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-04-25T04:38:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149869 2024-04-25T04:38:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Snapshot ID: 3282 2024-04-25T04:38:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 150044 2024-04-25T04:38:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 150200 2024-04-25T04:38:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 3 2024-04-25T04:38:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-04-25T04:38:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 150375 2024-04-25T04:38:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:38:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:38:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 148981 2024-04-25T04:41:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149115 2024-04-25T04:41:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149249 2024-04-25T04:41:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149384 2024-04-25T04:41:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:26.444 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:26.445 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149539 2024-04-25T04:41:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:27.842 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149715 2024-04-25T04:41:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 149870 2024-04-25T04:41:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 150045 2024-04-25T04:41:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 150201 2024-04-25T04:41:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-04-25T04:41:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 150376 2024-04-25T04:41:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ret=124 2024-04-25T04:41:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( ret != 124 )) 2024-04-25T04:41:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:41:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 1 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd0 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary1 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary1 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary1 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 2 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd1 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary2 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary2 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary2 2024-04-25T04:41:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 3 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd2 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary3 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary3 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary3 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 4 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd3 2024-04-25T04:41:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary4 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary4 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary4 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 5 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd4 2024-04-25T04:41:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary5 2024-04-25T04:41:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary5 2024-04-25T04:41:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary5 2024-04-25T04:41:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 6 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 7 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd5 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary6 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary6 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary6 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd6 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary7 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary7 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary7 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 8 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ COMPARE_PIDS+=($!) 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156355 2024-04-25T04:41:33.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 10 2024-04-25T04:41:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd9 2024-04-25T04:41:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary10 2024-04-25T04:41:33.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary10 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary10 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd7 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary8 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary8 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary8 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ compare_demoted_promoted_image 9 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local dev=/dev/rbd8 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local img=image-primary9 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local mntpt=test-primary9 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local demote_md5 promote_md5 2024-04-25T04:41:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo umount test-primary9 2024-04-25T04:41:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:41:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:41:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:41:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:41:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:41:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:41:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:41:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:41:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:41:34.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:41:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:41:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=5117eac9ad2e143745b3c1481c195a68 2024-04-25T04:42:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-04-25T04:42:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary5 2024-04-25T04:42:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-04-25T04:42:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=ce88e188e029b7634d55d9a9101dab11 2024-04-25T04:42:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-04-25T04:42:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=ebf09ce7d036b84f9f985cbf2cd54e03 2024-04-25T04:42:29.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-04-25T04:42:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=b29077efa4524d405fd036eca648135e 2024-04-25T04:42:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-04-25T04:42:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary4 2024-04-25T04:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-04-25T04:42:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary7 2024-04-25T04:42:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-04-25T04:42:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary2 2024-04-25T04:42:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-04-25T04:42:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=9d595e06bcbad747399661aa2c5d345e 2024-04-25T04:42:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-04-25T04:42:30.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary6 2024-04-25T04:42:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:42:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=681c19a9408cd883c3584c7b0f94fdf2 2024-04-25T04:42:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-04-25T04:42:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=de4a3b596c071186475940e71db96f8f 2024-04-25T04:42:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-04-25T04:42:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:42:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:42:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=f73b9fbc1b52259a2e7776c79d448d44 2024-04-25T04:42:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-04-25T04:42:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=229f61e56a408f4d80a38de1894deb98 2024-04-25T04:42:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-04-25T04:42:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary3 2024-04-25T04:42:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-04-25T04:42:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary8 2024-04-25T04:42:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:31.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-04-25T04:42:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary10 2024-04-25T04:42:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-04-25T04:42:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary9 2024-04-25T04:42:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-04-25T04:42:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-04-25T04:42:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:31.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_md5=2f86ff9c9e83bb3191d90e9747f971d9 2024-04-25T04:42:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-04-25T04:42:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ demote_image cluster1 mirror image-primary1 2024-04-25T04:42:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:32.844 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-04-25T04:42:32.911 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:42:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:32.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:42:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:42:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:42:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:42:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-04-25T04:42:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:32.996 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-04-25T04:42:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-04-25T04:42:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:42:33.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:42:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:42:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ff549457-25a5-4b41-922c-5c7a9dcb33a4 2024-04-25T04:42:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4a0ff156-3598-4982-991b-a69c3615b291 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:42:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8e3fcaef-3fd6-4c33-9da0-5f6e4433df39 2024-04-25T04:42:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:33.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:42:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:42:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:33.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:42:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:34.020 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:42:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:42:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 35ac40cd-f116-41a1-9df2-1315e256f174 2024-04-25T04:42:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:34.081 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:42:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f89850dc-5bb7-460e-9d8e-c7d8d0b23af6 2024-04-25T04:42:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1a6cb9ad-5b91-4698-af22-d148427b9c03 2024-04-25T04:42:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:42:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 0e2cdce8-7612-40bd-84f8-e6b644244c10 2024-04-25T04:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:42:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:42:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:35.119 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:42:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:42:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:42:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ff549457-25a5-4b41-922c-5c7a9dcb33a4 2024-04-25T04:42:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:42:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8e3fcaef-3fd6-4c33-9da0-5f6e4433df39 2024-04-25T04:42:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:35.218 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:42:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4a0ff156-3598-4982-991b-a69c3615b291 2024-04-25T04:42:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image demoted to non-primary 2024-04-25T04:42:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-04-25T04:42:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:42:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:42:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6c3ebfcc-0796-4a1c-8804-a70579d0f02e 2024-04-25T04:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:42:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:42:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 35ac40cd-f116-41a1-9df2-1315e256f174 2024-04-25T04:42:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:42:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f89850dc-5bb7-460e-9d8e-c7d8d0b23af6 2024-04-25T04:42:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:42:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1a6cb9ad-5b91-4698-af22-d148427b9c03 2024-04-25T04:42:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:36.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:36.180 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 0e2cdce8-7612-40bd-84f8-e6b644244c10 2024-04-25T04:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:36.193 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:42:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:42:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ec2399e4-6cb9-4060-91b4-d0aab89b8fa7 2024-04-25T04:42:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 2 2024-04-25T04:42:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:42:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:42:37.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:42:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:42:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:37.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:42:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:42:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:42:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6c3ebfcc-0796-4a1c-8804-a70579d0f02e 2024-04-25T04:42:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:42:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:42:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ec2399e4-6cb9-4060-91b4-d0aab89b8fa7 2024-04-25T04:42:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:39.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 4 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:42:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:42:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:42:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:42:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:42:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ff549457-25a5-4b41-922c-5c7a9dcb33a4 2024-04-25T04:42:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:39.298 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:42:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8e3fcaef-3fd6-4c33-9da0-5f6e4433df39 2024-04-25T04:42:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:39.300 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:42:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4a0ff156-3598-4982-991b-a69c3615b291 2024-04-25T04:42:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:42:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:42:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:40.184 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:42:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:42:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 35ac40cd-f116-41a1-9df2-1315e256f174 2024-04-25T04:42:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f89850dc-5bb7-460e-9d8e-c7d8d0b23af6 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1a6cb9ad-5b91-4698-af22-d148427b9c03 2024-04-25T04:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:42:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 0e2cdce8-7612-40bd-84f8-e6b644244c10 2024-04-25T04:42:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:40.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:42:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:42:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:42:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6c3ebfcc-0796-4a1c-8804-a70579d0f02e 2024-04-25T04:42:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:42:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:42:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ec2399e4-6cb9-4060-91b4-d0aab89b8fa7 2024-04-25T04:42:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:42:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:42:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:42:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:42:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+stopped 2024-04-25T04:42:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: local image is primary 2024-04-25T04:42:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:16' 2024-04-25T04:42:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 1 2024-04-25T04:42:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 8 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:42:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:42:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ff549457-25a5-4b41-922c-5c7a9dcb33a4 2024-04-25T04:42:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:47.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:42:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8e3fcaef-3fd6-4c33-9da0-5f6e4433df39 2024-04-25T04:42:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:47.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4a0ff156-3598-4982-991b-a69c3615b291 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:42:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:47.601 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:42:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:42:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:42:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:42:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-04-25T04:42:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:42:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:42:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f89850dc-5bb7-460e-9d8e-c7d8d0b23af6 2024-04-25T04:42:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:42:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 35ac40cd-f116-41a1-9df2-1315e256f174 2024-04-25T04:42:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1a6cb9ad-5b91-4698-af22-d148427b9c03 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:48.371 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 0e2cdce8-7612-40bd-84f8-e6b644244c10 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:48.382 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.384 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:42:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:42:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:42:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:42:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-04-25T04:42:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:48.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:42:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:42:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:42:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:42:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-04-25T04:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-04-25T04:42:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-04-25T04:42:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:42:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ff549457-25a5-4b41-922c-5c7a9dcb33a4 2024-04-25T04:42:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:42:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4a0ff156-3598-4982-991b-a69c3615b291 2024-04-25T04:42:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8e3fcaef-3fd6-4c33-9da0-5f6e4433df39 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:42:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:42:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:42:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:42:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:42:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:48.928 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary7 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-04-25T04:42:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:42:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary2 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-04-25T04:42:48.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary4 2024-04-25T04:42:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:42:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-04-25T04:42:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-04-25T04:42:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-04-25T04:42:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-04-25T04:42:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd2 2024-04-25T04:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd0 2024-04-25T04:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd2 2024-04-25T04:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd1 2024-04-25T04:42:49.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd0 2024-04-25T04:42:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd1 2024-04-25T04:42:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-04-25T04:42:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-04-25T04:42:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-04-25T04:42:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-04-25T04:42:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-04-25T04:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-04-25T04:42:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:42:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f89850dc-5bb7-460e-9d8e-c7d8d0b23af6 2024-04-25T04:42:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:42:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 35ac40cd-f116-41a1-9df2-1315e256f174 2024-04-25T04:42:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1a6cb9ad-5b91-4698-af22-d148427b9c03 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.713 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 0e2cdce8-7612-40bd-84f8-e6b644244c10 2024-04-25T04:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:49.730 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:49.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:42:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:42:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:42:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:42:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary10 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:50.024 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-04-25T04:42:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:42:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary3 2024-04-25T04:42:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:42:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-04-25T04:42:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:42:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary9 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-04-25T04:42:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:42:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary6 2024-04-25T04:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-04-25T04:42:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-04-25T04:42:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:50.204 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:42:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:42:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6c3ebfcc-0796-4a1c-8804-a70579d0f02e 2024-04-25T04:42:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:42:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:42:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-04-25T04:42:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:51.205 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:42:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-04-25T04:42:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-04-25T04:42:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:51.262 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-04-25T04:42:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-04-25T04:42:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:42:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ec2399e4-6cb9-4060-91b4-d0aab89b8fa7 2024-04-25T04:42:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:51.360 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-04-25T04:42:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:51.440 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-04-25T04:42:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd5 2024-04-25T04:42:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd5 2024-04-25T04:42:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd3 2024-04-25T04:42:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd4 2024-04-25T04:42:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd4 2024-04-25T04:42:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd3 2024-04-25T04:42:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd6 2024-04-25T04:42:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd6 2024-04-25T04:42:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:42:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6c3ebfcc-0796-4a1c-8804-a70579d0f02e 2024-04-25T04:42:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:42:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:42:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:51.745 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-04-25T04:42:51.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:42:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:42:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary1 2024-04-25T04:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-04-25T04:42:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:52.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-04-25T04:42:52.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd7 2024-04-25T04:42:52.556 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd7 2024-04-25T04:42:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-04-25T04:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ec2399e4-6cb9-4060-91b4-d0aab89b8fa7 2024-04-25T04:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:42:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:42:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:53.355 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:53.356 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary8 2024-04-25T04:42:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:53.357 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-04-25T04:42:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:42:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:53.358 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-04-25T04:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:53.359 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:42:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:42:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:42:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:53.580 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:53.581 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:53.612 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:53.615 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:42:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary41944233mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:42:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:42:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-04-25T04:42:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:42:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:42:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-04-25T04:42:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd8 2024-04-25T04:42:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd8 2024-04-25T04:42:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:42:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-04-25T04:42:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:42:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:42:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:54.973 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-04-25T04:42:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:42:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:42:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+unknown 2024-04-25T04:42:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: remote image is not primary 2024-04-25T04:42:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:42:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:42:46' 2024-04-25T04:42:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+unknown' 2024-04-25T04:42:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+unknown 2024-04-25T04:42:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:42:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: remote image is not primary 2024-04-25T04:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:42:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:42:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:42:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:42:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:42:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-04-25 04:42:46' 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+unknown 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:42:46' 2024-04-25T04:42:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:42:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:42:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-04-25T04:42:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1042024244mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+unknownremote image is not primary42024244mirror.0smithi1882024-04-25 04:42:46' 2024-04-25T04:42:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:42:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:42:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:42:46' 2024-04-25T04:42:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:42:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+unknown 2024-04-25T04:42:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+unknown 2024-04-25T04:42:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+unknown 2024-04-25T04:42:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+unknown 2024-04-25T04:42:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:42:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_image cluster2 mirror image-primary5 2024-04-25T04:42:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:42:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:42:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:42:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local force= 2024-04-25T04:42:55.596 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-04-25T04:42:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:Image promoted to primary 2024-04-25T04:42:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \n\b\d ]] 2024-04-25T04:42:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ krbd == \k\r\b\d ]] 2024-04-25T04:42:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-04-25T04:42:56.744 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ dev=/dev/rbd9 2024-04-25T04:42:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sudo md5sum /dev/rbd9 2024-04-25T04:42:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ awk '{print $1}' 2024-04-25T04:43:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=ebf09ce7d036b84f9f985cbf2cd54e03 2024-04-25T04:43:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-04-25T04:43:45.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=ce88e188e029b7634d55d9a9101dab11 2024-04-25T04:43:45.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-04-25T04:43:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ ce88e188e029b7634d55d9a9101dab11 != \c\e\8\8\e\1\8\8\e\0\2\9\b\7\6\3\4\d\5\5\d\9\a\9\1\0\1\d\a\b\1\1 ]] 2024-04-25T04:43:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ ebf09ce7d036b84f9f985cbf2cd54e03 != \e\b\f\0\9\c\e\7\d\0\3\6\b\8\4\f\9\f\9\8\5\c\b\f\2\c\d\5\4\e\0\3 ]] 2024-04-25T04:43:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=9d595e06bcbad747399661aa2c5d345e 2024-04-25T04:43:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-04-25T04:43:46.684 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 9d595e06bcbad747399661aa2c5d345e != \9\d\5\9\5\e\0\6\b\c\b\a\d\7\4\7\3\9\9\6\6\1\a\a\2\c\5\d\3\4\5\e ]] 2024-04-25T04:43:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=b29077efa4524d405fd036eca648135e 2024-04-25T04:43:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-04-25T04:43:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ b29077efa4524d405fd036eca648135e != \b\2\9\0\7\7\e\f\a\4\5\2\4\d\4\0\5\f\d\0\3\6\e\c\a\6\4\8\1\3\5\e ]] 2024-04-25T04:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=229f61e56a408f4d80a38de1894deb98 2024-04-25T04:43:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-04-25T04:43:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 229f61e56a408f4d80a38de1894deb98 != \2\2\9\f\6\1\e\5\6\a\4\0\8\f\4\d\8\0\a\3\8\d\e\1\8\9\4\d\e\b\9\8 ]] 2024-04-25T04:43:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=de4a3b596c071186475940e71db96f8f 2024-04-25T04:43:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-04-25T04:43:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ de4a3b596c071186475940e71db96f8f != \d\e\4\a\3\b\5\9\6\c\0\7\1\1\8\6\4\7\5\9\4\0\e\7\1\d\b\9\6\f\8\f ]] 2024-04-25T04:43:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=f73b9fbc1b52259a2e7776c79d448d44 2024-04-25T04:43:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-04-25T04:43:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=2f86ff9c9e83bb3191d90e9747f971d9 2024-04-25T04:43:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-04-25T04:43:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ f73b9fbc1b52259a2e7776c79d448d44 != \f\7\3\b\9\f\b\c\1\b\5\2\2\5\9\a\2\e\7\7\7\6\c\7\9\d\4\4\8\d\4\4 ]] 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 2f86ff9c9e83bb3191d90e9747f971d9 != \2\f\8\6\f\f\9\c\9\e\8\3\b\b\3\1\9\1\d\9\0\e\9\7\4\7\f\9\7\1\d\9 ]] 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156356 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156358 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156360 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156362 2024-04-25T04:43:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=681c19a9408cd883c3584c7b0f94fdf2 2024-04-25T04:43:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-04-25T04:43:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 681c19a9408cd883c3584c7b0f94fdf2 != \6\8\1\c\1\9\a\9\4\0\8\c\d\8\8\3\c\3\5\8\4\c\7\b\0\f\9\4\f\d\f\2 ]] 2024-04-25T04:43:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ promote_md5=5117eac9ad2e143745b3c1481c195a68 2024-04-25T04:43:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-04-25T04:43:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ [[ 5117eac9ad2e143745b3c1481c195a68 != \5\1\1\7\e\a\c\9\a\d\2\e\1\4\3\7\4\5\b\3\c\1\4\8\1\c\1\9\5\a\6\8 ]] 2024-04-25T04:43:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156364 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156366 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156367 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156370 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait 156371 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ (( RET != 0 )) 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary1 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:43:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:43:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:43:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:43:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-04-25T04:43:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary1: 2024-04-25T04:43:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 6c3ebfcc-0796-4a1c-8804-a70579d0f02e 2024-04-25T04:43:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:43:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020172,"remote_snapshot_timestamp":1714020172,"replay_state":"idle"} 2024-04-25T04:43:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:43:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:43:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:43:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:43:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:43:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020172,"remote_snapshot_timestamp":1714020172,"replay_state":"idle"} 2024-04-25T04:43:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:43:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:43:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:43:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:43:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:43:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:43:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:43:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:43:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:43:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:43:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:43:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING1041944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary16c3ebfcc-0796-4a1c-8804-a70579d0f02eup+replayingreplaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020172,"remote_snapshot_timestamp":1714020172,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+replayingreplaying, {"bytes_per_second":3679846.4,"bytes_per_snapshot":73596928.0,"last_snapshot_bytes":73596928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+replayingreplaying, {"bytes_per_second":3608985.6,"bytes_per_snapshot":72179712.0,"last_snapshot_bytes":72179712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+replayingreplaying, {"bytes_per_second":3607756.8,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+replayingreplaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020176,"remote_snapshot_timestamp":1714020176,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+replayingreplaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+replayingreplaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:43:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:43:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-04-25T04:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:43:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-04-25T04:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:43:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:43:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:43:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:43:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary1 2024-04-25T04:43:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:43:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:43:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-04-25T04:43:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-04-25T04:43:52.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:43:52.035+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667702000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:43:52.039 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:43:52.035+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cda280 [3/6c3ebfcc-0796-4a1c-8804-a70579d0f02e] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:43:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:43:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-04-25T04:43:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:53.479 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary1 2024-04-25T04:43:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:43:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:43:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary1 2024-04-25T04:43:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary1 2024-04-25T04:43:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary2 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:43:53.558 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:43:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:43:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-04-25T04:43:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:43:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:43:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:43:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:43:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:43:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-04-25T04:43:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary2: 2024-04-25T04:43:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 8e3fcaef-3fd6-4c33-9da0-5f6e4433df39 2024-04-25T04:43:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:43:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3679846.4,"bytes_per_snapshot":73596928.0,"last_snapshot_bytes":73596928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"} 2024-04-25T04:43:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:43:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:43:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:43:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:43:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:43:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3679846.4,"bytes_per_snapshot":73596928.0,"last_snapshot_bytes":73596928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"} 2024-04-25T04:43:54.643 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:43:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:43:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:43:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:43:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:43:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:43:54.649 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:43:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:43:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:43:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:43:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:43:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING941944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary28e3fcaef-3fd6-4c33-9da0-5f6e4433df39up+replayingreplaying, {"bytes_per_second":3679846.4,"bytes_per_snapshot":73596928.0,"last_snapshot_bytes":73596928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+replayingreplaying, {"bytes_per_second":3608985.6,"bytes_per_snapshot":72179712.0,"last_snapshot_bytes":72179712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+replayingreplaying, {"bytes_per_second":3607756.8,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+replayingreplaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020176,"remote_snapshot_timestamp":1714020176,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+replayingreplaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+replayingreplaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:43:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:43:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-04-25T04:43:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:43:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-04-25T04:43:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:43:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:43:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:43:54.774 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:43:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary2 2024-04-25T04:43:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:43:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:43:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-04-25T04:43:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-04-25T04:43:54.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:43:54.991+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596645636c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:43:54.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:43:54.991+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb180 [3/8e3fcaef-3fd6-4c33-9da0-5f6e4433df39] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:43:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:43:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-04-25T04:43:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary2 2024-04-25T04:43:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:43:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:43:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary2 2024-04-25T04:43:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary2 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary3 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:43:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:43:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:43:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:43:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:43:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-04-25T04:43:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:43:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:43:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:43:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:43:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:43:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-04-25T04:43:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary3: 2024-04-25T04:43:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: f89850dc-5bb7-460e-9d8e-c7d8d0b23af6 2024-04-25T04:43:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:43:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3608985.6,"bytes_per_snapshot":72179712.0,"last_snapshot_bytes":72179712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:43:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:43:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:43:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:43:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:43:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:43:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3608985.6,"bytes_per_snapshot":72179712.0,"last_snapshot_bytes":72179712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:43:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:43:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:43:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:43:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:43:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:43:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:43:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:43:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:43:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:43:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:43:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:43:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING841944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary3f89850dc-5bb7-460e-9d8e-c7d8d0b23af6up+replayingreplaying, {"bytes_per_second":3608985.6,"bytes_per_snapshot":72179712.0,"last_snapshot_bytes":72179712,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+replayingreplaying, {"bytes_per_second":3607756.8,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+replayingreplaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020176,"remote_snapshot_timestamp":1714020176,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+replayingreplaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+replayingreplaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:43:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:43:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-04-25T04:43:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:43:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-04-25T04:43:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:43:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:43:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:43:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary3 2024-04-25T04:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:43:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-04-25T04:43:57.841 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-04-25T04:43:57.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:43:57.987+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596654f1040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:43:57.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:43:57.987+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559668508a00 [3/f89850dc-5bb7-460e-9d8e-c7d8d0b23af6] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:43:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-04-25T04:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary3 2024-04-25T04:43:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:43:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:43:59.884 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary3 2024-04-25T04:43:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:43:59.943 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary3 2024-04-25T04:43:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:43:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary4 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:43:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:43:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:43:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:44:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-04-25T04:44:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:00.949 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:44:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:00.950 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-04-25T04:44:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary4: 2024-04-25T04:44:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 4a0ff156-3598-4982-991b-a69c3615b291 2024-04-25T04:44:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:44:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3607756.8,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"} 2024-04-25T04:44:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:44:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:44:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:44:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:44:01.029 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:44:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3607756.8,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"} 2024-04-25T04:44:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:44:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:44:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:44:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:44:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:44:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:44:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:44:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING741944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary44a0ff156-3598-4982-991b-a69c3615b291up+replayingreplaying, {"bytes_per_second":3607756.8,"bytes_per_snapshot":72155136.0,"last_snapshot_bytes":72155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+replayingreplaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020176,"remote_snapshot_timestamp":1714020176,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+replayingreplaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+replayingreplaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:44:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:44:01.139 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-04-25T04:44:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:01.144 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-04-25T04:44:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:44:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:44:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary4 2024-04-25T04:44:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:44:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-04-25T04:44:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-04-25T04:44:01.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:01.360+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664791380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:01.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:01.360+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508780 [3/4a0ff156-3598-4982-991b-a69c3615b291] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:44:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-04-25T04:44:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary4 2024-04-25T04:44:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:44:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary4 2024-04-25T04:44:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary4 2024-04-25T04:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary5 2024-04-25T04:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-04-25T04:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:44:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-04-25T04:44:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:44:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-04-25T04:44:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary5: 2024-04-25T04:44:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 79b73870-cdaf-4a02-8ef1-9eacfbc402aa 2024-04-25T04:44:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:44:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020176,"remote_snapshot_timestamp":1714020176,"replay_state":"idle"} 2024-04-25T04:44:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:44:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:44:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:44:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:44:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:44:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020176,"remote_snapshot_timestamp":1714020176,"replay_state":"idle"} 2024-04-25T04:44:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:44:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:44:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:44:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:44:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:44:04.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:44:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING641944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary579b73870-cdaf-4a02-8ef1-9eacfbc402aaup+replayingreplaying, {"bytes_per_second":3642982.4,"bytes_per_snapshot":72859648.0,"last_snapshot_bytes":72859648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020176,"remote_snapshot_timestamp":1714020176,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+replayingreplaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+replayingreplaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:44:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:44:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-04-25T04:44:04.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:04.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-04-25T04:44:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:44:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:44:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary5 2024-04-25T04:44:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:44:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-04-25T04:44:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-04-25T04:44:04.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:04.381+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596654f0d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:04.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:04.381+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668508000 [3/79b73870-cdaf-4a02-8ef1-9eacfbc402aa] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:44:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-04-25T04:44:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary5 2024-04-25T04:44:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:44:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary5 2024-04-25T04:44:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary5 2024-04-25T04:44:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:44:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary6 2024-04-25T04:44:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-04-25T04:44:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:44:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:44:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-04-25T04:44:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:44:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:44:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-04-25T04:44:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:44:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-04-25T04:44:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary6: 2024-04-25T04:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 0e2cdce8-7612-40bd-84f8-e6b644244c10 2024-04-25T04:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:44:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:44:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:44:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:44:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:44:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:44:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:07.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:44:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:07.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:44:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:44:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:44:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:44:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:44:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:44:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:44:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:44:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING541944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary60e2cdce8-7612-40bd-84f8-e6b644244c10up+replayingreplaying, {"bytes_per_second":3608166.4,"bytes_per_snapshot":72163328.0,"last_snapshot_bytes":72163328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+replayingreplaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:44:07.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:44:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-04-25T04:44:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-04-25T04:44:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:44:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:44:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary6 2024-04-25T04:44:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:44:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-04-25T04:44:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-04-25T04:44:07.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:07.466+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5596652f7a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:07.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:07.466+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668211680 [3/0e2cdce8-7612-40bd-84f8-e6b644244c10] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-04-25T04:44:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary6 2024-04-25T04:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary6 2024-04-25T04:44:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary6 2024-04-25T04:44:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:44:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary7 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-04-25T04:44:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary7: 2024-04-25T04:44:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ff549457-25a5-4b41-922c-5c7a9dcb33a4 2024-04-25T04:44:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:44:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"} 2024-04-25T04:44:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:44:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:44:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:44:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:44:10.047 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:44:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"} 2024-04-25T04:44:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:44:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:44:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:44:10.055 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:44:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:44:10.057 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:44:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:44:10.149 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING441944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary7ff549457-25a5-4b41-922c-5c7a9dcb33a4up+replayingreplaying, {"bytes_per_second":3667148.8,"bytes_per_snapshot":73342976.0,"last_snapshot_bytes":73342976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020169,"remote_snapshot_timestamp":1714020169,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:44:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:44:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-04-25T04:44:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-04-25T04:44:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:44:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:44:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary7 2024-04-25T04:44:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:44:10.163 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-04-25T04:44:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-04-25T04:44:10.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:10.370+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c5ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:10.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:10.370+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559668211400 [3/ff549457-25a5-4b41-922c-5c7a9dcb33a4] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:44:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-04-25T04:44:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:11.925 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary7 2024-04-25T04:44:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:44:11.926 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary7 2024-04-25T04:44:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary7 2024-04-25T04:44:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:44:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary8 2024-04-25T04:44:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-04-25T04:44:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:44:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-04-25T04:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary8: 2024-04-25T04:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: ec2399e4-6cb9-4060-91b4-d0aab89b8fa7 2024-04-25T04:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"} 2024-04-25T04:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:44:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:44:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:44:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:44:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"} 2024-04-25T04:44:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:44:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:44:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:13.069 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:44:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:44:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:44:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:44:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:44:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:44:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:44:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:44:13.164 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING341944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary8ec2399e4-6cb9-4060-91b4-d0aab89b8fa7up+replayingreplaying, {"bytes_per_second":3693363.2,"bytes_per_snapshot":73867264.0,"last_snapshot_bytes":73867264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020174,"remote_snapshot_timestamp":1714020174,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:44:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:44:13.165 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-04-25T04:44:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-04-25T04:44:13.175 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:44:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:44:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary8 2024-04-25T04:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-04-25T04:44:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-04-25T04:44:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:13.391+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559664c57380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:13.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:13.391+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdb680 [3/ec2399e4-6cb9-4060-91b4-d0aab89b8fa7] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:14.924 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:44:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-04-25T04:44:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary8 2024-04-25T04:44:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:44:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary8 2024-04-25T04:44:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary8 2024-04-25T04:44:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:44:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary9 2024-04-25T04:44:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-04-25T04:44:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:14.986 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:44:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:15.988 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-04-25T04:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary9: 2024-04-25T04:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 1a6cb9ad-5b91-4698-af22-d148427b9c03 2024-04-25T04:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:43:46' 2024-04-25T04:44:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:44:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:44:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:44:16.109 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:44:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:44:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:44:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:44:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:44:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:44:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-04-25 04:43:46' 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:43:46' 2024-04-25T04:44:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:44:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING241944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":3253862.4,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46image-primary91a6cb9ad-5b91-4698-af22-d148427b9c03up+replayingreplaying, {"bytes_per_second":3103744.0,"bytes_per_snapshot":62074880.0,"last_snapshot_bytes":62074880,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:43:46' 2024-04-25T04:44:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:44:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-04-25T04:44:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:43:46' 2024-04-25T04:44:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-04-25T04:44:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:44:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:16.251 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:44:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary9 2024-04-25T04:44:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:44:16.254 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-04-25T04:44:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-04-25T04:44:16.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:16.554+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667a42d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:16.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:16.554+0000 7fb93a000700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/1a6cb9ad-5b91-4698-af22-d148427b9c03] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:16.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:16.814+0000 7fb931e00700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559667a42ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:16.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:16.814+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdbb80 [3/1a6cb9ad-5b91-4698-af22-d148427b9c03] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:44:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-04-25T04:44:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary9 2024-04-25T04:44:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:44:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary9 2024-04-25T04:44:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary9 2024-04-25T04:44:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for j in {1..10} 2024-04-25T04:44:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ IMG=image-primary10 2024-04-25T04:44:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-04-25T04:44:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' snapshot = journal ']' 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ sleep 1 2024-04-25T04:44:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-04-25T04:44:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:44:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:19.003 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local description_pattern= 2024-04-25T04:44:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local service_pattern= 2024-04-25T04:44:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-04-25T04:44:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='image-primary10: 2024-04-25T04:44:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: global_id: 35ac40cd-f116-41a1-9df2-1315e256f174 2024-04-25T04:44:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: state: up+replaying 2024-04-25T04:44:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:44:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: service: mirror.0 on smithi188 2024-04-25T04:44:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: last_update: 2024-04-25 04:44:16' 2024-04-25T04:44:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ state: .*up+replaying' 2024-04-25T04:44:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: state: up+replaying 2024-04-25T04:44:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^ description: .*' 2024-04-25T04:44:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"} 2024-04-25T04:44:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -n '' ']' 2024-04-25T04:44:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep '^up+' 2024-04-25T04:44:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep 'service: *mirror..* on ' 2024-04-25T04:44:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout: service: mirror.0 on smithi188 2024-04-25T04:44:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update 2024-04-25T04:44:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-04-25T04:44:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:44:16' 2024-04-25T04:44:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-04-25 04:44:16' 2024-04-25T04:44:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:44:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local state_pattern=up+replaying 2024-04-25T04:44:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local 'prev_last_update=2024-04-25 04:44:16' 2024-04-25T04:44:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local status 2024-04-25T04:44:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ CEPH_ARGS= 2024-04-25T04:44:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-04-25T04:44:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ status='WARNINGOKWARNING141944233mirror.0smithi18819.0.0-2424-gb98aab6ctrueOKimage-primary1035ac40cd-f116-41a1-9df2-1315e256f174up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65077248.0,"last_snapshot_bytes":65077248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1714020171,"remote_snapshot_timestamp":1714020171,"replay_state":"idle"}41944233mirror.0smithi1882024-04-25 04:44:16' 2024-04-25T04:44:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local last_update state 2024-04-25T04:44:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-04-25T04:44:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ last_update='2024-04-25 04:44:16' 2024-04-25T04:44:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-04-25T04:44:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ state=up+replaying 2024-04-25T04:44:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo up+replaying 2024-04-25T04:44:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep up+replaying 2024-04-25T04:44:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:up+replaying 2024-04-25T04:44:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ remove_image cluster2 mirror image-primary10 2024-04-25T04:44:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:44:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-04-25T04:44:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-04-25T04:44:19.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:19.415+0000 7fb932800700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55966810d040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-04-25T04:44:19.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi188.stderr:2024-04-25T04:44:19.415+0000 7fb93aa00700 -1 rbd::mirror::ImageReplayer: 0x559664cdaf00 [3/35ac40cd-f116-41a1-9df2-1315e256f174] operator(): start failed: (85) Interrupted system call should be restarted 2024-04-25T04:44:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-04-25T04:44:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-04-25T04:44:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local pool=mirror 2024-04-25T04:44:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local image=image-primary10 2024-04-25T04:44:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-04-25T04:44:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rbd --cluster cluster1 ls mirror 2024-04-25T04:44:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ grep -wq image-primary10 2024-04-25T04:44:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 0 2024-04-25T04:44:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -rf test-primary10 2024-04-25T04:44:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo OK 2024-04-25T04:44:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:OK 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ cleanup 0 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local error_code=0 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ set +e 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 -ne 0 ']' 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -z '' ']' 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster instance 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T04:44:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-25T04:44:22.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:22.721+0000 7f919f000700 -1 rbd::mirror::image_deleter::SnapshotPurgeRequest: 0x55907392f320 handle_snap_remove: failed to remove snapshot: (2) No such file or directory 2024-04-25T04:44:22.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:22.725+0000 7f917de00700 -1 rbd::mirror::image_deleter::TrashRemoveRequest: 0x559072bbbc30 handle_purge_snapshots: failed to purge image snapshots: (2) No such file or directory 2024-04-25T04:44:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror' does not exist 2024-04-25T04:44:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T04:44:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-04-25T04:44:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror' does not exist 2024-04-25T04:44:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T04:44:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-25T04:44:24.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.557+0000 7f91a7c00700 -1 rbd::mirror::LeaderWatcher: 0x55907124a000 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-04-25T04:44:24.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.557+0000 7f919fa00700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5590711a8480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-25T04:44:24.559 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.557+0000 7f919fa00700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55907144c480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-25T04:44:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.557+0000 7f919fa00700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5590711a8b40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-04-25T04:44:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.558+0000 7f919fa00700 -1 rbd::mirror::PoolWatcher: 0x5590701123c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-25T04:44:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.558+0000 7f919fa00700 -1 rbd::mirror::PoolWatcher: 0x5590712c4000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-25T04:44:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.558+0000 7f919fa00700 -1 rbd::mirror::PoolWatcher: 0x5590712c41e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-25T04:44:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.558+0000 7f9197800700 -1 rbd::mirror::PoolWatcher: 0x5590712c4960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-25T04:44:24.560 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.558+0000 7f9197800700 -1 rbd::mirror::PoolWatcher: 0x5590712c4780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-25T04:44:24.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.558+0000 7f9197800700 -1 rbd::mirror::PoolWatcher: 0x5590712c4d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-04-25T04:44:24.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.559+0000 7f91a7200700 -1 rbd::mirror::LeaderWatcher: 0x55907124a000 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-04-25T04:44:24.561 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi188.stderr:2024-04-25T04:44:24.559+0000 7f91a7200700 -1 rbd::mirror::LeaderWatcher: 0x55907124a000 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-04-25T04:44:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror_parent' does not exist 2024-04-25T04:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ CEPH_ARGS= 2024-04-25T04:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-04-25T04:44:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:pool 'mirror_parent' does not exist 2024-04-25T04:44:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-25T04:44:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ stop_mirrors cluster1 2024-04-25T04:44:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1 2024-04-25T04:44:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local sig= 2024-04-25T04:44:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ seq 0 0 2024-04-25T04:44:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ stop_mirror cluster1:0 '' 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster1:0 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local sig= 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test -n 1 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ stop_mirrors cluster2 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local sig= 2024-04-25T04:44:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:++ seq 0 0 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ stop_mirror cluster2:0 '' 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local cluster=cluster2:0 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ local sig= 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test -n 1 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ return 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' -z 1 ']' 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ test '' = /tmp/tmp.gg15hFy7Os 2024-04-25T04:44:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ rm -Rf /tmp/tmp.gg15hFy7Os 2024-04-25T04:44:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ '[' 0 -eq 0 ']' 2024-04-25T04:44:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ echo OK 2024-04-25T04:44:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stdout:OK 2024-04-25T04:44:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi188.stderr:+ exit 0 2024-04-25T04:44:25.899 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T04:44:25.899 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-04-25T04:44:26.213 INFO:tasks.workunit:Stopping ['rbd/compare_mirror_images.sh'] on cluster1.client.mirror... 2024-04-25T04:44:26.213 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-04-25T04:44:27.304 DEBUG:teuthology.parallel:result is None 2024-04-25T04:44:27.305 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-25T04:44:27.377 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-25T04:44:27.378 DEBUG:teuthology.orchestra.run.smithi188:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-04-25T04:44:27.422 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-04-25T04:44:27.423 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-25T04:44:27.432 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-04-25T04:44:27.433 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:44:27.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-04-25T04:44:27.488 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-04-25T04:44:27.500 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-04-25T04:44:27.500 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:44:27.550 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-04-25T04:44:27.550 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T04:44:27.560 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-04-25T04:44:27.560 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-25T04:44:28.057 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T04:44:28.057 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-04-25T04:44:28.075 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":4345,"stamp":"2024-04-25T04:44:27.118756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":235,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2561536,"kb_used_data":19000,"kb_used_omap":4,"kb_used_meta":2542459,"kb_avail":280553984,"statfs":{"total":289910292480,"available":287287279616,"internally_reserved":0,"allocated":19456000,"data_stored":11270883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2603478351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-465107090,"num_objects":-178,"num_object_clones":-13,"num_object_copies":-356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-178,"num_whiteouts":0,"num_read":-649887,"num_read_kb":-36889035,"num_write":-134254,"num_write_kb":-29413033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-4202512,"num_keys_recovered":-7,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.941530"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.199034+0000","last_change":"2024-04-25T03:22:40.043790+0000","last_active":"2024-04-25T04:44:21.199034+0000","last_peered":"2024-04-25T04:44:21.199034+0000","last_clean":"2024-04-25T04:44:21.199034+0000","last_became_active":"2024-04-25T03:22:40.043485+0000","last_became_peered":"2024-04-25T03:22:40.043485+0000","last_unstale":"2024-04-25T04:44:21.199034+0000","last_undegraded":"2024-04-25T04:44:21.199034+0000","last_fullsized":"2024-04-25T04:44:21.199034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:16.009245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3794,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.180443+0000","last_change":"2024-04-25T03:22:40.041527+0000","last_active":"2024-04-25T04:44:21.180443+0000","last_peered":"2024-04-25T04:44:21.180443+0000","last_clean":"2024-04-25T04:44:21.180443+0000","last_became_active":"2024-04-25T03:22:40.041252+0000","last_became_peered":"2024-04-25T03:22:40.041252+0000","last_unstale":"2024-04-25T04:44:21.180443+0000","last_undegraded":"2024-04-25T04:44:21.180443+0000","last_fullsized":"2024-04-25T04:44:21.180443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.379837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3799,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717082+0000","last_change":"2024-04-25T03:22:40.041941+0000","last_active":"2024-04-25T04:44:23.717082+0000","last_peered":"2024-04-25T04:44:23.717082+0000","last_clean":"2024-04-25T04:44:23.717082+0000","last_became_active":"2024-04-25T03:22:40.041506+0000","last_became_peered":"2024-04-25T03:22:40.041506+0000","last_unstale":"2024-04-25T04:44:23.717082+0000","last_undegraded":"2024-04-25T04:44:23.717082+0000","last_fullsized":"2024-04-25T04:44:23.717082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:06.230110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.197442+0000","last_change":"2024-04-25T03:22:40.038756+0000","last_active":"2024-04-25T04:44:21.197442+0000","last_peered":"2024-04-25T04:44:21.197442+0000","last_clean":"2024-04-25T04:44:21.197442+0000","last_became_active":"2024-04-25T03:22:40.038422+0000","last_became_peered":"2024-04-25T03:22:40.038422+0000","last_unstale":"2024-04-25T04:44:21.197442+0000","last_undegraded":"2024-04-25T04:44:21.197442+0000","last_fullsized":"2024-04-25T04:44:21.197442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:30.846205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":3929,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.192242+0000","last_change":"2024-04-25T03:22:35.092603+0000","last_active":"2024-04-25T04:44:21.192242+0000","last_peered":"2024-04-25T04:44:21.192242+0000","last_clean":"2024-04-25T04:44:21.192242+0000","last_became_active":"2024-04-25T03:22:35.092211+0000","last_became_peered":"2024-04-25T03:22:35.092211+0000","last_unstale":"2024-04-25T04:44:21.192242+0000","last_undegraded":"2024-04-25T04:44:21.192242+0000","last_fullsized":"2024-04-25T04:44:21.192242+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_clean_scrub_stamp":"2024-04-25T03:22:34.001265+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:44.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3800,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717557+0000","last_change":"2024-04-25T03:22:40.041323+0000","last_active":"2024-04-25T04:44:23.717557+0000","last_peered":"2024-04-25T04:44:23.717557+0000","last_clean":"2024-04-25T04:44:23.717557+0000","last_became_active":"2024-04-25T03:22:40.040953+0000","last_became_peered":"2024-04-25T03:22:40.040953+0000","last_unstale":"2024-04-25T04:44:23.717557+0000","last_undegraded":"2024-04-25T04:44:23.717557+0000","last_fullsized":"2024-04-25T04:44:23.717557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:05:08.093470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.197906+0000","last_change":"2024-04-25T03:22:40.044306+0000","last_active":"2024-04-25T04:44:21.197906+0000","last_peered":"2024-04-25T04:44:21.197906+0000","last_clean":"2024-04-25T04:44:21.197906+0000","last_became_active":"2024-04-25T03:22:40.044067+0000","last_became_peered":"2024-04-25T03:22:40.044067+0000","last_unstale":"2024-04-25T04:44:21.197906+0000","last_undegraded":"2024-04-25T04:44:21.197906+0000","last_fullsized":"2024-04-25T04:44:21.197906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:15.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3793,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.200315+0000","last_change":"2024-04-25T03:22:40.040386+0000","last_active":"2024-04-25T04:44:21.200315+0000","last_peered":"2024-04-25T04:44:21.200315+0000","last_clean":"2024-04-25T04:44:21.200315+0000","last_became_active":"2024-04-25T03:22:40.040101+0000","last_became_peered":"2024-04-25T03:22:40.040101+0000","last_unstale":"2024-04-25T04:44:21.200315+0000","last_undegraded":"2024-04-25T04:44:21.200315+0000","last_fullsized":"2024-04-25T04:44:21.200315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:26.847462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3793,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.201296+0000","last_change":"2024-04-25T03:22:40.043916+0000","last_active":"2024-04-25T04:44:21.201296+0000","last_peered":"2024-04-25T04:44:21.201296+0000","last_clean":"2024-04-25T04:44:21.201296+0000","last_became_active":"2024-04-25T03:22:40.043563+0000","last_became_peered":"2024-04-25T03:22:40.043563+0000","last_unstale":"2024-04-25T04:44:21.201296+0000","last_undegraded":"2024-04-25T04:44:21.201296+0000","last_fullsized":"2024-04-25T04:44:21.201296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:20.327339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673945,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":872100,"kb_used_data":6520,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":93499740,"statfs":{"total":96636764160,"available":95743733760,"internally_reserved":0,"allocated":6676480,"data_stored":3950929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706649,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763220,"kb_used_data":5960,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":93608620,"statfs":{"total":96636764160,"available":95855226880,"internally_reserved":0,"allocated":6103040,"data_stored":3369025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673945,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926216,"kb_used_data":6520,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":93445624,"statfs":{"total":96636764160,"available":95688318976,"internally_reserved":0,"allocated":6676480,"data_stored":3950929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:44:28.075 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-25T04:44:28.483 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T04:44:28.486 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-04-25T04:44:28.503 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":4345,"stamp":"2024-04-25T04:44:27.118756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":235,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2561536,"kb_used_data":19000,"kb_used_omap":4,"kb_used_meta":2542459,"kb_avail":280553984,"statfs":{"total":289910292480,"available":287287279616,"internally_reserved":0,"allocated":19456000,"data_stored":11270883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2603478351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-465107090,"num_objects":-178,"num_object_clones":-13,"num_object_copies":-356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-178,"num_whiteouts":0,"num_read":-649887,"num_read_kb":-36889035,"num_write":-134254,"num_write_kb":-29413033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-4202512,"num_keys_recovered":-7,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.941530"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.199034+0000","last_change":"2024-04-25T03:22:40.043790+0000","last_active":"2024-04-25T04:44:21.199034+0000","last_peered":"2024-04-25T04:44:21.199034+0000","last_clean":"2024-04-25T04:44:21.199034+0000","last_became_active":"2024-04-25T03:22:40.043485+0000","last_became_peered":"2024-04-25T03:22:40.043485+0000","last_unstale":"2024-04-25T04:44:21.199034+0000","last_undegraded":"2024-04-25T04:44:21.199034+0000","last_fullsized":"2024-04-25T04:44:21.199034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:16.009245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3794,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.180443+0000","last_change":"2024-04-25T03:22:40.041527+0000","last_active":"2024-04-25T04:44:21.180443+0000","last_peered":"2024-04-25T04:44:21.180443+0000","last_clean":"2024-04-25T04:44:21.180443+0000","last_became_active":"2024-04-25T03:22:40.041252+0000","last_became_peered":"2024-04-25T03:22:40.041252+0000","last_unstale":"2024-04-25T04:44:21.180443+0000","last_undegraded":"2024-04-25T04:44:21.180443+0000","last_fullsized":"2024-04-25T04:44:21.180443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.379837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3799,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717082+0000","last_change":"2024-04-25T03:22:40.041941+0000","last_active":"2024-04-25T04:44:23.717082+0000","last_peered":"2024-04-25T04:44:23.717082+0000","last_clean":"2024-04-25T04:44:23.717082+0000","last_became_active":"2024-04-25T03:22:40.041506+0000","last_became_peered":"2024-04-25T03:22:40.041506+0000","last_unstale":"2024-04-25T04:44:23.717082+0000","last_undegraded":"2024-04-25T04:44:23.717082+0000","last_fullsized":"2024-04-25T04:44:23.717082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:06.230110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.197442+0000","last_change":"2024-04-25T03:22:40.038756+0000","last_active":"2024-04-25T04:44:21.197442+0000","last_peered":"2024-04-25T04:44:21.197442+0000","last_clean":"2024-04-25T04:44:21.197442+0000","last_became_active":"2024-04-25T03:22:40.038422+0000","last_became_peered":"2024-04-25T03:22:40.038422+0000","last_unstale":"2024-04-25T04:44:21.197442+0000","last_undegraded":"2024-04-25T04:44:21.197442+0000","last_fullsized":"2024-04-25T04:44:21.197442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:30.846205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":3929,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.192242+0000","last_change":"2024-04-25T03:22:35.092603+0000","last_active":"2024-04-25T04:44:21.192242+0000","last_peered":"2024-04-25T04:44:21.192242+0000","last_clean":"2024-04-25T04:44:21.192242+0000","last_became_active":"2024-04-25T03:22:35.092211+0000","last_became_peered":"2024-04-25T03:22:35.092211+0000","last_unstale":"2024-04-25T04:44:21.192242+0000","last_undegraded":"2024-04-25T04:44:21.192242+0000","last_fullsized":"2024-04-25T04:44:21.192242+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_clean_scrub_stamp":"2024-04-25T03:22:34.001265+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:44.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3800,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717557+0000","last_change":"2024-04-25T03:22:40.041323+0000","last_active":"2024-04-25T04:44:23.717557+0000","last_peered":"2024-04-25T04:44:23.717557+0000","last_clean":"2024-04-25T04:44:23.717557+0000","last_became_active":"2024-04-25T03:22:40.040953+0000","last_became_peered":"2024-04-25T03:22:40.040953+0000","last_unstale":"2024-04-25T04:44:23.717557+0000","last_undegraded":"2024-04-25T04:44:23.717557+0000","last_fullsized":"2024-04-25T04:44:23.717557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:05:08.093470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.197906+0000","last_change":"2024-04-25T03:22:40.044306+0000","last_active":"2024-04-25T04:44:21.197906+0000","last_peered":"2024-04-25T04:44:21.197906+0000","last_clean":"2024-04-25T04:44:21.197906+0000","last_became_active":"2024-04-25T03:22:40.044067+0000","last_became_peered":"2024-04-25T03:22:40.044067+0000","last_unstale":"2024-04-25T04:44:21.197906+0000","last_undegraded":"2024-04-25T04:44:21.197906+0000","last_fullsized":"2024-04-25T04:44:21.197906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:15.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3793,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.200315+0000","last_change":"2024-04-25T03:22:40.040386+0000","last_active":"2024-04-25T04:44:21.200315+0000","last_peered":"2024-04-25T04:44:21.200315+0000","last_clean":"2024-04-25T04:44:21.200315+0000","last_became_active":"2024-04-25T03:22:40.040101+0000","last_became_peered":"2024-04-25T03:22:40.040101+0000","last_unstale":"2024-04-25T04:44:21.200315+0000","last_undegraded":"2024-04-25T04:44:21.200315+0000","last_fullsized":"2024-04-25T04:44:21.200315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:26.847462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3793,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.201296+0000","last_change":"2024-04-25T03:22:40.043916+0000","last_active":"2024-04-25T04:44:21.201296+0000","last_peered":"2024-04-25T04:44:21.201296+0000","last_clean":"2024-04-25T04:44:21.201296+0000","last_became_active":"2024-04-25T03:22:40.043563+0000","last_became_peered":"2024-04-25T03:22:40.043563+0000","last_unstale":"2024-04-25T04:44:21.201296+0000","last_undegraded":"2024-04-25T04:44:21.201296+0000","last_fullsized":"2024-04-25T04:44:21.201296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:20.327339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673945,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":872100,"kb_used_data":6520,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":93499740,"statfs":{"total":96636764160,"available":95743733760,"internally_reserved":0,"allocated":6676480,"data_stored":3950929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706649,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763220,"kb_used_data":5960,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":93608620,"statfs":{"total":96636764160,"available":95855226880,"internally_reserved":0,"allocated":6103040,"data_stored":3369025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673945,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926216,"kb_used_data":6520,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":93445624,"statfs":{"total":96636764160,"available":95688318976,"internally_reserved":0,"allocated":6676480,"data_stored":3950929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:44:28.504 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-04-25T04:44:28.504 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-25T04:44:28.912 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T04:44:28.916 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-04-25T04:44:28.933 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":4345,"stamp":"2024-04-25T04:44:27.118756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":235,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2561536,"kb_used_data":19000,"kb_used_omap":4,"kb_used_meta":2542459,"kb_avail":280553984,"statfs":{"total":289910292480,"available":287287279616,"internally_reserved":0,"allocated":19456000,"data_stored":11270883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2603478351},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-465107090,"num_objects":-178,"num_object_clones":-13,"num_object_copies":-356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-178,"num_whiteouts":0,"num_read":-649887,"num_read_kb":-36889035,"num_write":-134254,"num_write_kb":-29413033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-4202512,"num_keys_recovered":-7,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.941530"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.199034+0000","last_change":"2024-04-25T03:22:40.043790+0000","last_active":"2024-04-25T04:44:21.199034+0000","last_peered":"2024-04-25T04:44:21.199034+0000","last_clean":"2024-04-25T04:44:21.199034+0000","last_became_active":"2024-04-25T03:22:40.043485+0000","last_became_peered":"2024-04-25T03:22:40.043485+0000","last_unstale":"2024-04-25T04:44:21.199034+0000","last_undegraded":"2024-04-25T04:44:21.199034+0000","last_fullsized":"2024-04-25T04:44:21.199034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:16.009245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3794,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.180443+0000","last_change":"2024-04-25T03:22:40.041527+0000","last_active":"2024-04-25T04:44:21.180443+0000","last_peered":"2024-04-25T04:44:21.180443+0000","last_clean":"2024-04-25T04:44:21.180443+0000","last_became_active":"2024-04-25T03:22:40.041252+0000","last_became_peered":"2024-04-25T03:22:40.041252+0000","last_unstale":"2024-04-25T04:44:21.180443+0000","last_undegraded":"2024-04-25T04:44:21.180443+0000","last_fullsized":"2024-04-25T04:44:21.180443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.379837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3799,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717082+0000","last_change":"2024-04-25T03:22:40.041941+0000","last_active":"2024-04-25T04:44:23.717082+0000","last_peered":"2024-04-25T04:44:23.717082+0000","last_clean":"2024-04-25T04:44:23.717082+0000","last_became_active":"2024-04-25T03:22:40.041506+0000","last_became_peered":"2024-04-25T03:22:40.041506+0000","last_unstale":"2024-04-25T04:44:23.717082+0000","last_undegraded":"2024-04-25T04:44:23.717082+0000","last_fullsized":"2024-04-25T04:44:23.717082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:06.230110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.197442+0000","last_change":"2024-04-25T03:22:40.038756+0000","last_active":"2024-04-25T04:44:21.197442+0000","last_peered":"2024-04-25T04:44:21.197442+0000","last_clean":"2024-04-25T04:44:21.197442+0000","last_became_active":"2024-04-25T03:22:40.038422+0000","last_became_peered":"2024-04-25T03:22:40.038422+0000","last_unstale":"2024-04-25T04:44:21.197442+0000","last_undegraded":"2024-04-25T04:44:21.197442+0000","last_fullsized":"2024-04-25T04:44:21.197442+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:30.846205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":3929,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.192242+0000","last_change":"2024-04-25T03:22:35.092603+0000","last_active":"2024-04-25T04:44:21.192242+0000","last_peered":"2024-04-25T04:44:21.192242+0000","last_clean":"2024-04-25T04:44:21.192242+0000","last_became_active":"2024-04-25T03:22:35.092211+0000","last_became_peered":"2024-04-25T03:22:35.092211+0000","last_unstale":"2024-04-25T04:44:21.192242+0000","last_undegraded":"2024-04-25T04:44:21.192242+0000","last_fullsized":"2024-04-25T04:44:21.192242+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_clean_scrub_stamp":"2024-04-25T03:22:34.001265+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:44.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3800,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717557+0000","last_change":"2024-04-25T03:22:40.041323+0000","last_active":"2024-04-25T04:44:23.717557+0000","last_peered":"2024-04-25T04:44:23.717557+0000","last_clean":"2024-04-25T04:44:23.717557+0000","last_became_active":"2024-04-25T03:22:40.040953+0000","last_became_peered":"2024-04-25T03:22:40.040953+0000","last_unstale":"2024-04-25T04:44:23.717557+0000","last_undegraded":"2024-04-25T04:44:23.717557+0000","last_fullsized":"2024-04-25T04:44:23.717557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:05:08.093470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3795,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.197906+0000","last_change":"2024-04-25T03:22:40.044306+0000","last_active":"2024-04-25T04:44:21.197906+0000","last_peered":"2024-04-25T04:44:21.197906+0000","last_clean":"2024-04-25T04:44:21.197906+0000","last_became_active":"2024-04-25T03:22:40.044067+0000","last_became_peered":"2024-04-25T03:22:40.044067+0000","last_unstale":"2024-04-25T04:44:21.197906+0000","last_undegraded":"2024-04-25T04:44:21.197906+0000","last_fullsized":"2024-04-25T04:44:21.197906+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:15.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3793,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.200315+0000","last_change":"2024-04-25T03:22:40.040386+0000","last_active":"2024-04-25T04:44:21.200315+0000","last_peered":"2024-04-25T04:44:21.200315+0000","last_clean":"2024-04-25T04:44:21.200315+0000","last_became_active":"2024-04-25T03:22:40.040101+0000","last_became_peered":"2024-04-25T03:22:40.040101+0000","last_unstale":"2024-04-25T04:44:21.200315+0000","last_undegraded":"2024-04-25T04:44:21.200315+0000","last_fullsized":"2024-04-25T04:44:21.200315+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:26.847462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3793,"reported_epoch":1889,"state":"active+clean","last_fresh":"2024-04-25T04:44:21.201296+0000","last_change":"2024-04-25T03:22:40.043916+0000","last_active":"2024-04-25T04:44:21.201296+0000","last_peered":"2024-04-25T04:44:21.201296+0000","last_clean":"2024-04-25T04:44:21.201296+0000","last_became_active":"2024-04-25T03:22:40.043563+0000","last_became_peered":"2024-04-25T03:22:40.043563+0000","last_unstale":"2024-04-25T04:44:21.201296+0000","last_undegraded":"2024-04-25T04:44:21.201296+0000","last_fullsized":"2024-04-25T04:44:21.201296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:20.327339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673945,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":872100,"kb_used_data":6520,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":93499740,"statfs":{"total":96636764160,"available":95743733760,"internally_reserved":0,"allocated":6676480,"data_stored":3950929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706649,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763220,"kb_used_data":5960,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":93608620,"statfs":{"total":96636764160,"available":95855226880,"internally_reserved":0,"allocated":6103040,"data_stored":3369025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673945,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":926216,"kb_used_data":6520,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":93445624,"statfs":{"total":96636764160,"available":95688318976,"internally_reserved":0,"allocated":6676480,"data_stored":3950929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:44:28.934 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-04-25T04:44:29.346 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T04:44:29.346 INFO:teuthology.orchestra.run.smithi188.stdout:{"epoch":1892,"fsid":"deb0a81e-61a8-4bcb-b28f-c9ae781a23c3","created":"2024-04-25T03:22:21.673300+0000","modified":"2024-04-25T04:44:25.693761+0000","last_up_change":"2024-04-25T03:22:31.986774+0000","last_in_change":"2024-04-25T03:22:23.083689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:22:33.530774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T03:22:38.170609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d6d5e21d-e5ed-4729-bed8-061304dc8eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6816","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6817","nonce":3218143884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6818","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6819","nonce":3218143884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6822","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6823","nonce":3218143884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6820","nonce":3218143884},{"type":"v1","addr":"172.21.15.188:6821","nonce":3218143884}]},"public_addr":"172.21.15.188:6817/3218143884","cluster_addr":"172.21.15.188:6819/3218143884","heartbeat_back_addr":"172.21.15.188:6823/3218143884","heartbeat_front_addr":"172.21.15.188:6821/3218143884","state":["exists","up"]},{"osd":1,"uuid":"ad829940-0ef2-4ba3-acf5-58a3f5149282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6808","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6809","nonce":3431739252}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6810","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6811","nonce":3431739252}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6814","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6815","nonce":3431739252}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6812","nonce":3431739252},{"type":"v1","addr":"172.21.15.188:6813","nonce":3431739252}]},"public_addr":"172.21.15.188:6809/3431739252","cluster_addr":"172.21.15.188:6811/3431739252","heartbeat_back_addr":"172.21.15.188:6815/3431739252","heartbeat_front_addr":"172.21.15.188:6813/3431739252","state":["exists","up"]},{"osd":2,"uuid":"eb229790-f353-4d08-be57-b1bffedf8966","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6800","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6801","nonce":840963968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6802","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6803","nonce":840963968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6806","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6807","nonce":840963968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.188:6804","nonce":840963968},{"type":"v1","addr":"172.21.15.188:6805","nonce":840963968}]},"public_addr":"172.21.15.188:6801/840963968","cluster_addr":"172.21.15.188:6803/840963968","heartbeat_back_addr":"172.21.15.188:6807/840963968","heartbeat_front_addr":"172.21.15.188:6805/840963968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.458734+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.452573+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:22:28.448465+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3238,"length":1},{"begin":3243,"length":1},{"begin":3245,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T04:44:30.369 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-04-25T04:44:30.369 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T04:44:30.619 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-25T04:44:30.620 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T04:44:30.620 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-25T04:44:30.637 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-04-25T04:44:31.054 INFO:teuthology.orchestra.run.smithi188.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T04:44:31.073 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-04-25T04:44:31.073 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T04:44:31.295 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-25T04:44:31.295 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T04:44:31.295 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-25T04:44:31.314 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-04-25T04:44:31.735 INFO:teuthology.orchestra.run.smithi188.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T04:44:31.752 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-04-25T04:44:31.752 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T04:44:31.972 INFO:teuthology.orchestra.run.smithi188.stdout:{ 2024-04-25T04:44:31.972 INFO:teuthology.orchestra.run.smithi188.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T04:44:31.972 INFO:teuthology.orchestra.run.smithi188.stdout:} 2024-04-25T04:44:31.990 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-04-25T04:44:32.407 INFO:teuthology.orchestra.run.smithi188.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T04:44:32.425 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-25T04:44:32.838 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T04:44:32.841 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-04-25T04:44:32.858 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":4347,"stamp":"2024-04-25T04:44:31.119354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2579608,"kb_used_data":19056,"kb_used_omap":4,"kb_used_meta":2560507,"kb_avail":280535912,"statfs":{"total":289910292480,"available":287268773888,"internally_reserved":0,"allocated":19513344,"data_stored":11295153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2621959503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":-650194,"num_read_kb":-36889303,"num_write":-134455,"num_write_kb":-29413055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-4202512,"num_keys_recovered":-7,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001368"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3801,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:25.705299+0000","last_change":"2024-04-25T03:22:40.043790+0000","last_active":"2024-04-25T04:44:25.705299+0000","last_peered":"2024-04-25T04:44:25.705299+0000","last_clean":"2024-04-25T04:44:25.705299+0000","last_became_active":"2024-04-25T03:22:40.043485+0000","last_became_peered":"2024-04-25T03:22:40.043485+0000","last_unstale":"2024-04-25T04:44:25.705299+0000","last_undegraded":"2024-04-25T04:44:25.705299+0000","last_fullsized":"2024-04-25T04:44:25.705299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:16.009245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3800,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.004921+0000","last_change":"2024-04-25T03:22:40.041527+0000","last_active":"2024-04-25T04:44:26.004921+0000","last_peered":"2024-04-25T04:44:26.004921+0000","last_clean":"2024-04-25T04:44:26.004921+0000","last_became_active":"2024-04-25T03:22:40.041252+0000","last_became_peered":"2024-04-25T03:22:40.041252+0000","last_unstale":"2024-04-25T04:44:26.004921+0000","last_undegraded":"2024-04-25T04:44:26.004921+0000","last_fullsized":"2024-04-25T04:44:26.004921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.379837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3799,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717082+0000","last_change":"2024-04-25T03:22:40.041941+0000","last_active":"2024-04-25T04:44:23.717082+0000","last_peered":"2024-04-25T04:44:23.717082+0000","last_clean":"2024-04-25T04:44:23.717082+0000","last_became_active":"2024-04-25T03:22:40.041506+0000","last_became_peered":"2024-04-25T03:22:40.041506+0000","last_unstale":"2024-04-25T04:44:23.717082+0000","last_undegraded":"2024-04-25T04:44:23.717082+0000","last_fullsized":"2024-04-25T04:44:23.717082+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:06.230110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3801,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:25.702648+0000","last_change":"2024-04-25T03:22:40.038756+0000","last_active":"2024-04-25T04:44:25.702648+0000","last_peered":"2024-04-25T04:44:25.702648+0000","last_clean":"2024-04-25T04:44:25.702648+0000","last_became_active":"2024-04-25T03:22:40.038422+0000","last_became_peered":"2024-04-25T03:22:40.038422+0000","last_unstale":"2024-04-25T04:44:25.702648+0000","last_undegraded":"2024-04-25T04:44:25.702648+0000","last_fullsized":"2024-04-25T04:44:25.702648+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:21:30.846205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":3935,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:25.700607+0000","last_change":"2024-04-25T03:22:35.092603+0000","last_active":"2024-04-25T04:44:25.700607+0000","last_peered":"2024-04-25T04:44:25.700607+0000","last_clean":"2024-04-25T04:44:25.700607+0000","last_became_active":"2024-04-25T03:22:35.092211+0000","last_became_peered":"2024-04-25T03:22:35.092211+0000","last_unstale":"2024-04-25T04:44:25.700607+0000","last_undegraded":"2024-04-25T04:44:25.700607+0000","last_fullsized":"2024-04-25T04:44:25.700607+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:34.001265+0000","last_clean_scrub_stamp":"2024-04-25T03:22:34.001265+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:39:44.961911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3800,"reported_epoch":1891,"state":"active+clean","last_fresh":"2024-04-25T04:44:23.717557+0000","last_change":"2024-04-25T03:22:40.041323+0000","last_active":"2024-04-25T04:44:23.717557+0000","last_peered":"2024-04-25T04:44:23.717557+0000","last_clean":"2024-04-25T04:44:23.717557+0000","last_became_active":"2024-04-25T03:22:40.040953+0000","last_became_peered":"2024-04-25T03:22:40.040953+0000","last_unstale":"2024-04-25T04:44:23.717557+0000","last_undegraded":"2024-04-25T04:44:23.717557+0000","last_fullsized":"2024-04-25T04:44:23.717557+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:05:08.093470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3801,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:25.705353+0000","last_change":"2024-04-25T03:22:40.044306+0000","last_active":"2024-04-25T04:44:25.705353+0000","last_peered":"2024-04-25T04:44:25.705353+0000","last_clean":"2024-04-25T04:44:25.705353+0000","last_became_active":"2024-04-25T03:22:40.044067+0000","last_became_peered":"2024-04-25T03:22:40.044067+0000","last_unstale":"2024-04-25T04:44:25.705353+0000","last_undegraded":"2024-04-25T04:44:25.705353+0000","last_fullsized":"2024-04-25T04:44:25.705353+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:15.549389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3799,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:25.706786+0000","last_change":"2024-04-25T03:22:40.040386+0000","last_active":"2024-04-25T04:44:25.706786+0000","last_peered":"2024-04-25T04:44:25.706786+0000","last_clean":"2024-04-25T04:44:25.706786+0000","last_became_active":"2024-04-25T03:22:40.040101+0000","last_became_peered":"2024-04-25T03:22:40.040101+0000","last_unstale":"2024-04-25T04:44:25.706786+0000","last_undegraded":"2024-04-25T04:44:25.706786+0000","last_fullsized":"2024-04-25T04:44:25.706786+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:57:26.847462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3799,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:25.708562+0000","last_change":"2024-04-25T03:22:40.043916+0000","last_active":"2024-04-25T04:44:25.708562+0000","last_peered":"2024-04-25T04:44:25.708562+0000","last_clean":"2024-04-25T04:44:25.708562+0000","last_became_active":"2024-04-25T03:22:40.043563+0000","last_became_peered":"2024-04-25T03:22:40.043563+0000","last_unstale":"2024-04-25T04:44:25.708562+0000","last_undegraded":"2024-04-25T04:44:25.708562+0000","last_fullsized":"2024-04-25T04:44:25.708562+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:22:39.017226+0000","last_clean_scrub_stamp":"2024-04-25T03:22:39.017226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:20.327339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673946,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":872096,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":93499744,"statfs":{"total":96636764160,"available":95743737856,"internally_reserved":0,"allocated":6705152,"data_stored":3963064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706649,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763220,"kb_used_data":5960,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":93608620,"statfs":{"total":96636764160,"available":95855226880,"internally_reserved":0,"allocated":6103040,"data_stored":3369025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673946,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944292,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":93427548,"statfs":{"total":96636764160,"available":95669809152,"internally_reserved":0,"allocated":6705152,"data_stored":3963064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:44:32.862 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.862 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.862 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.863 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.863 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T03:22:34.001265+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=34, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.863 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.863 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.863 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.863 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T03:22:39.017226+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=22, tm_sec=39, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=44, tm_sec=29, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:44:32.863 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T04:44:52.864 DEBUG:teuthology.orchestra.run.smithi188:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-04-25T04:44:53.285 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-04-25T04:44:53.288 INFO:teuthology.orchestra.run.smithi188.stderr:dumped all 2024-04-25T04:44:53.306 INFO:teuthology.orchestra.run.smithi188.stdout:{"pg_ready":true,"pg_map":{"version":4358,"stamp":"2024-04-25T04:44:53.124116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2579600,"kb_used_data":19064,"kb_used_omap":4,"kb_used_meta":2560507,"kb_avail":280535920,"statfs":{"total":289910292480,"available":287268782080,"internally_reserved":0,"allocated":19521536,"data_stored":11298824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":2621959503},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002562"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3810,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:35.240056+0000","last_change":"2024-04-25T04:44:35.240078+0000","last_active":"2024-04-25T04:44:35.240056+0000","last_peered":"2024-04-25T04:44:35.240056+0000","last_clean":"2024-04-25T04:44:35.240056+0000","last_became_active":"2024-04-25T03:22:40.043485+0000","last_became_peered":"2024-04-25T03:22:40.043485+0000","last_unstale":"2024-04-25T04:44:35.240056+0000","last_undegraded":"2024-04-25T04:44:35.240056+0000","last_fullsized":"2024-04-25T04:44:35.240056+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:35.239951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:35.239951+0000","last_clean_scrub_stamp":"2024-04-25T04:44:35.239951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:52:00.714057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3809,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:31.595642+0000","last_change":"2024-04-25T04:44:31.595697+0000","last_active":"2024-04-25T04:44:31.595642+0000","last_peered":"2024-04-25T04:44:31.595642+0000","last_clean":"2024-04-25T04:44:31.595642+0000","last_became_active":"2024-04-25T03:22:40.041252+0000","last_became_peered":"2024-04-25T03:22:40.041252+0000","last_unstale":"2024-04-25T04:44:31.595642+0000","last_undegraded":"2024-04-25T04:44:31.595642+0000","last_fullsized":"2024-04-25T04:44:31.595642+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:31.595482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:31.595482+0000","last_clean_scrub_stamp":"2024-04-25T04:44:31.595482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:00:30.873351+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3810,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:33.811165+0000","last_change":"2024-04-25T04:44:33.811203+0000","last_active":"2024-04-25T04:44:33.811165+0000","last_peered":"2024-04-25T04:44:33.811165+0000","last_clean":"2024-04-25T04:44:33.811165+0000","last_became_active":"2024-04-25T03:22:40.041506+0000","last_became_peered":"2024-04-25T03:22:40.041506+0000","last_unstale":"2024-04-25T04:44:33.811165+0000","last_undegraded":"2024-04-25T04:44:33.811165+0000","last_fullsized":"2024-04-25T04:44:33.811165+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:33.811050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:33.811050+0000","last_clean_scrub_stamp":"2024-04-25T04:44:33.811050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:41:50.961541+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3810,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:33.256900+0000","last_change":"2024-04-25T04:44:33.256936+0000","last_active":"2024-04-25T04:44:33.256900+0000","last_peered":"2024-04-25T04:44:33.256900+0000","last_clean":"2024-04-25T04:44:33.256900+0000","last_became_active":"2024-04-25T03:22:40.038422+0000","last_became_peered":"2024-04-25T03:22:40.038422+0000","last_unstale":"2024-04-25T04:44:33.256900+0000","last_undegraded":"2024-04-25T04:44:33.256900+0000","last_fullsized":"2024-04-25T04:44:33.256900+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:33.256786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:33.256786+0000","last_clean_scrub_stamp":"2024-04-25T04:44:33.256786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:53:24.707581+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1892'99","reported_seq":3946,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:32.136438+0000","last_change":"2024-04-25T04:44:32.136470+0000","last_active":"2024-04-25T04:44:32.136438+0000","last_peered":"2024-04-25T04:44:32.136438+0000","last_clean":"2024-04-25T04:44:32.136438+0000","last_became_active":"2024-04-25T03:22:35.092211+0000","last_became_peered":"2024-04-25T03:22:35.092211+0000","last_unstale":"2024-04-25T04:44:32.136438+0000","last_undegraded":"2024-04-25T04:44:32.136438+0000","last_fullsized":"2024-04-25T04:44:32.136438+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"1892'99","last_scrub_stamp":"2024-04-25T04:44:32.136362+0000","last_deep_scrub":"1892'99","last_deep_scrub_stamp":"2024-04-25T04:44:32.136362+0000","last_clean_scrub_stamp":"2024-04-25T04:44:32.136362+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:03:49.058491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3811,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:32.782950+0000","last_change":"2024-04-25T04:44:32.783005+0000","last_active":"2024-04-25T04:44:32.782950+0000","last_peered":"2024-04-25T04:44:32.782950+0000","last_clean":"2024-04-25T04:44:32.782950+0000","last_became_active":"2024-04-25T03:22:40.040953+0000","last_became_peered":"2024-04-25T03:22:40.040953+0000","last_unstale":"2024-04-25T04:44:32.782950+0000","last_undegraded":"2024-04-25T04:44:32.782950+0000","last_fullsized":"2024-04-25T04:44:32.782950+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:32.782838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:32.782838+0000","last_clean_scrub_stamp":"2024-04-25T04:44:32.782838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:48:16.597238+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3810,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:37.241667+0000","last_change":"2024-04-25T04:44:37.241701+0000","last_active":"2024-04-25T04:44:37.241667+0000","last_peered":"2024-04-25T04:44:37.241667+0000","last_clean":"2024-04-25T04:44:37.241667+0000","last_became_active":"2024-04-25T03:22:40.044067+0000","last_became_peered":"2024-04-25T03:22:40.044067+0000","last_unstale":"2024-04-25T04:44:37.241667+0000","last_undegraded":"2024-04-25T04:44:37.241667+0000","last_fullsized":"2024-04-25T04:44:37.241667+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:37.241506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:37.241506+0000","last_clean_scrub_stamp":"2024-04-25T04:44:37.241506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:33.575516+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3808,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:34.239335+0000","last_change":"2024-04-25T04:44:34.239372+0000","last_active":"2024-04-25T04:44:34.239335+0000","last_peered":"2024-04-25T04:44:34.239335+0000","last_clean":"2024-04-25T04:44:34.239335+0000","last_became_active":"2024-04-25T03:22:40.040101+0000","last_became_peered":"2024-04-25T03:22:40.040101+0000","last_unstale":"2024-04-25T04:44:34.239335+0000","last_undegraded":"2024-04-25T04:44:34.239335+0000","last_fullsized":"2024-04-25T04:44:34.239335+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:34.239212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:34.239212+0000","last_clean_scrub_stamp":"2024-04-25T04:44:34.239212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:48:00.267337+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3808,"reported_epoch":1892,"state":"active+clean","last_fresh":"2024-04-25T04:44:36.286088+0000","last_change":"2024-04-25T04:44:36.286110+0000","last_active":"2024-04-25T04:44:36.286088+0000","last_peered":"2024-04-25T04:44:36.286088+0000","last_clean":"2024-04-25T04:44:36.286088+0000","last_became_active":"2024-04-25T03:22:40.043563+0000","last_became_peered":"2024-04-25T03:22:40.043563+0000","last_unstale":"2024-04-25T04:44:36.286088+0000","last_undegraded":"2024-04-25T04:44:36.286088+0000","last_fullsized":"2024-04-25T04:44:36.286088+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:44:36.285980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:44:36.285980+0000","last_clean_scrub_stamp":"2024-04-25T04:44:36.285980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:01:47.345098+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949673951,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":872092,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":93499748,"statfs":{"total":96636764160,"available":95743741952,"internally_reserved":0,"allocated":6705152,"data_stored":3963064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654706654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":763224,"kb_used_data":5968,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":93608616,"statfs":{"total":96636764160,"available":95855222784,"internally_reserved":0,"allocated":6111232,"data_stored":3372696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949673951,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":944284,"kb_used_data":6548,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":93427556,"statfs":{"total":96636764160,"available":95669817344,"internally_reserved":0,"allocated":6705152,"data_stored":3963064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:44:53.307 DEBUG:teuthology.orchestra.run.smithi188:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-04-25T04:44:53.739 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T04:44:53.739 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T04:44:53.739 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-04-25T04:44:53.740 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:44:57.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-04-25T04:44:59.842 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-04-25T04:44:59.842 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-04-25T04:44:59.843 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:03.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-04-25T04:45:03.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-04-25T04:45:05.945 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-04-25T04:45:05.945 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-04-25T04:45:05.945 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:08.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2024-04-25T04:45:08.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-04-25T04:45:12.046 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-04-25T04:45:12.047 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T04:45:12.047 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-04-25T04:45:12.047 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:12.131 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-04-25T04:45:12.131 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T04:45:12.131 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-04-25T04:45:12.132 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:12.169 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-04-25T04:45:12.169 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T04:45:12.170 DEBUG:teuthology.orchestra.run.smithi188:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-25T04:45:12.386 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T04:45:12.386 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-04-25T04:45:12.616 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T04:45:12.616 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-04-25T04:45:12.759 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T04:45:12.759 DEBUG:teuthology.orchestra.run.smithi188:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-04-25T04:45:12.914 INFO:tasks.ceph:Archiving mon data... 2024-04-25T04:45:12.916 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/data/cluster2.mon.a.tgz 2024-04-25T04:45:12.916 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-04-25T04:45:12.966 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.gj3REd9wAy 2024-04-25T04:45:12.967 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.gj3REd9wAy 2024-04-25T04:45:13.172 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.gj3REd9wAy 2024-04-25T04:45:13.317 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.gj3REd9wAy is 537KB 2024-04-25T04:45:13.418 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.gj3REd9wAy 2024-04-25T04:45:13.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2024-04-25T04:45:13.462 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T04:45:13.462 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T04:45:13.468 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T04:45:13.833 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T04:45:13.843 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-04-25T04:45:13.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-25T04:45:14.581 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T04:45:14.584 INFO:teuthology.orchestra.run.smithi173.stderr:dumped all 2024-04-25T04:45:14.601 INFO:teuthology.orchestra.run.smithi173.stdout:{"pg_ready":true,"pg_map":{"version":4402,"stamp":"2024-04-25T04:45:14.088751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3799748,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":3779387,"kb_avail":279315772,"statfs":{"total":289910292480,"available":286019350528,"internally_reserved":0,"allocated":20799488,"data_stored":11373122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":3870092623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002420"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.731051+0000","last_change":"2024-04-25T03:22:00.453221+0000","last_active":"2024-04-25T04:44:24.731051+0000","last_peered":"2024-04-25T04:44:24.731051+0000","last_clean":"2024-04-25T04:44:24.731051+0000","last_became_active":"2024-04-25T03:22:00.452902+0000","last_became_peered":"2024-04-25T03:22:00.452902+0000","last_unstale":"2024-04-25T04:44:24.731051+0000","last_undegraded":"2024-04-25T04:44:24.731051+0000","last_fullsized":"2024-04-25T04:44:24.731051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:35:05.745342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3833,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.083180+0000","last_change":"2024-04-25T03:22:00.452610+0000","last_active":"2024-04-25T04:44:26.083180+0000","last_peered":"2024-04-25T04:44:26.083180+0000","last_clean":"2024-04-25T04:44:26.083180+0000","last_became_active":"2024-04-25T03:22:00.452291+0000","last_became_peered":"2024-04-25T03:22:00.452291+0000","last_unstale":"2024-04-25T04:44:26.083180+0000","last_undegraded":"2024-04-25T04:44:26.083180+0000","last_fullsized":"2024-04-25T04:44:26.083180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:42:10.308241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3833,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.390015+0000","last_change":"2024-04-25T03:22:00.452201+0000","last_active":"2024-04-25T04:44:26.390015+0000","last_peered":"2024-04-25T04:44:26.390015+0000","last_clean":"2024-04-25T04:44:26.390015+0000","last_became_active":"2024-04-25T03:22:00.451642+0000","last_became_peered":"2024-04-25T03:22:00.451642+0000","last_unstale":"2024-04-25T04:44:26.390015+0000","last_undegraded":"2024-04-25T04:44:26.390015+0000","last_fullsized":"2024-04-25T04:44:26.390015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:31.160601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729721+0000","last_change":"2024-04-25T03:22:00.450555+0000","last_active":"2024-04-25T04:44:24.729721+0000","last_peered":"2024-04-25T04:44:24.729721+0000","last_clean":"2024-04-25T04:44:24.729721+0000","last_became_active":"2024-04-25T03:22:00.450263+0000","last_became_peered":"2024-04-25T03:22:00.450263+0000","last_unstale":"2024-04-25T04:44:24.729721+0000","last_undegraded":"2024-04-25T04:44:24.729721+0000","last_fullsized":"2024-04-25T04:44:24.729721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:21:57.478981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":3967,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729471+0000","last_change":"2024-04-25T03:21:54.133861+0000","last_active":"2024-04-25T04:44:24.729471+0000","last_peered":"2024-04-25T04:44:24.729471+0000","last_clean":"2024-04-25T04:44:24.729471+0000","last_became_active":"2024-04-25T03:21:54.133471+0000","last_became_peered":"2024-04-25T03:21:54.133471+0000","last_unstale":"2024-04-25T04:44:24.729471+0000","last_undegraded":"2024-04-25T04:44:24.729471+0000","last_fullsized":"2024-04-25T04:44:24.729471+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_clean_scrub_stamp":"2024-04-25T03:21:53.124515+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:53:23.257337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3835,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.388849+0000","last_change":"2024-04-25T03:22:00.451023+0000","last_active":"2024-04-25T04:44:26.388849+0000","last_peered":"2024-04-25T04:44:26.388849+0000","last_clean":"2024-04-25T04:44:26.388849+0000","last_became_active":"2024-04-25T03:22:00.450704+0000","last_became_peered":"2024-04-25T03:22:00.450704+0000","last_unstale":"2024-04-25T04:44:26.388849+0000","last_undegraded":"2024-04-25T04:44:26.388849+0000","last_fullsized":"2024-04-25T04:44:26.388849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:05.312724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.727707+0000","last_change":"2024-04-25T03:22:00.453301+0000","last_active":"2024-04-25T04:44:24.727707+0000","last_peered":"2024-04-25T04:44:24.727707+0000","last_clean":"2024-04-25T04:44:24.727707+0000","last_became_active":"2024-04-25T03:22:00.453022+0000","last_became_peered":"2024-04-25T03:22:00.453022+0000","last_unstale":"2024-04-25T04:44:24.727707+0000","last_undegraded":"2024-04-25T04:44:24.727707+0000","last_fullsized":"2024-04-25T04:44:24.727707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:38.378533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.726881+0000","last_change":"2024-04-25T03:22:00.454554+0000","last_active":"2024-04-25T04:44:24.726881+0000","last_peered":"2024-04-25T04:44:24.726881+0000","last_clean":"2024-04-25T04:44:24.726881+0000","last_became_active":"2024-04-25T03:22:00.454254+0000","last_became_peered":"2024-04-25T03:22:00.454254+0000","last_unstale":"2024-04-25T04:44:24.726881+0000","last_undegraded":"2024-04-25T04:44:24.726881+0000","last_fullsized":"2024-04-25T04:44:24.726881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:08.518908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729702+0000","last_change":"2024-04-25T03:22:00.454572+0000","last_active":"2024-04-25T04:44:24.729702+0000","last_peered":"2024-04-25T04:44:24.729702+0000","last_clean":"2024-04-25T04:44:24.729702+0000","last_became_active":"2024-04-25T03:22:00.454212+0000","last_became_peered":"2024-04-25T03:22:00.454212+0000","last_unstale":"2024-04-25T04:44:24.729702+0000","last_undegraded":"2024-04-25T04:44:24.729702+0000","last_fullsized":"2024-04-25T04:44:24.729702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:00.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1156540,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":1150142,"kb_avail":93215300,"statfs":{"total":96636764160,"available":95452467200,"internally_reserved":0,"allocated":6537216,"data_stored":3397462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177745861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1377860,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1370878,"kb_avail":92993980,"statfs":{"total":96636764160,"available":95225835520,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1403779525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1265348,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1258366,"kb_avail":93106492,"statfs":{"total":96636764160,"available":95341047808,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1288567237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:45:14.602 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-25T04:45:15.012 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T04:45:15.015 INFO:teuthology.orchestra.run.smithi173.stderr:dumped all 2024-04-25T04:45:15.033 INFO:teuthology.orchestra.run.smithi173.stdout:{"pg_ready":true,"pg_map":{"version":4402,"stamp":"2024-04-25T04:45:14.088751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3799748,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":3779387,"kb_avail":279315772,"statfs":{"total":289910292480,"available":286019350528,"internally_reserved":0,"allocated":20799488,"data_stored":11373122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":3870092623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002420"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.731051+0000","last_change":"2024-04-25T03:22:00.453221+0000","last_active":"2024-04-25T04:44:24.731051+0000","last_peered":"2024-04-25T04:44:24.731051+0000","last_clean":"2024-04-25T04:44:24.731051+0000","last_became_active":"2024-04-25T03:22:00.452902+0000","last_became_peered":"2024-04-25T03:22:00.452902+0000","last_unstale":"2024-04-25T04:44:24.731051+0000","last_undegraded":"2024-04-25T04:44:24.731051+0000","last_fullsized":"2024-04-25T04:44:24.731051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:35:05.745342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3833,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.083180+0000","last_change":"2024-04-25T03:22:00.452610+0000","last_active":"2024-04-25T04:44:26.083180+0000","last_peered":"2024-04-25T04:44:26.083180+0000","last_clean":"2024-04-25T04:44:26.083180+0000","last_became_active":"2024-04-25T03:22:00.452291+0000","last_became_peered":"2024-04-25T03:22:00.452291+0000","last_unstale":"2024-04-25T04:44:26.083180+0000","last_undegraded":"2024-04-25T04:44:26.083180+0000","last_fullsized":"2024-04-25T04:44:26.083180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:42:10.308241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3833,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.390015+0000","last_change":"2024-04-25T03:22:00.452201+0000","last_active":"2024-04-25T04:44:26.390015+0000","last_peered":"2024-04-25T04:44:26.390015+0000","last_clean":"2024-04-25T04:44:26.390015+0000","last_became_active":"2024-04-25T03:22:00.451642+0000","last_became_peered":"2024-04-25T03:22:00.451642+0000","last_unstale":"2024-04-25T04:44:26.390015+0000","last_undegraded":"2024-04-25T04:44:26.390015+0000","last_fullsized":"2024-04-25T04:44:26.390015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:31.160601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729721+0000","last_change":"2024-04-25T03:22:00.450555+0000","last_active":"2024-04-25T04:44:24.729721+0000","last_peered":"2024-04-25T04:44:24.729721+0000","last_clean":"2024-04-25T04:44:24.729721+0000","last_became_active":"2024-04-25T03:22:00.450263+0000","last_became_peered":"2024-04-25T03:22:00.450263+0000","last_unstale":"2024-04-25T04:44:24.729721+0000","last_undegraded":"2024-04-25T04:44:24.729721+0000","last_fullsized":"2024-04-25T04:44:24.729721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:21:57.478981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":3967,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729471+0000","last_change":"2024-04-25T03:21:54.133861+0000","last_active":"2024-04-25T04:44:24.729471+0000","last_peered":"2024-04-25T04:44:24.729471+0000","last_clean":"2024-04-25T04:44:24.729471+0000","last_became_active":"2024-04-25T03:21:54.133471+0000","last_became_peered":"2024-04-25T03:21:54.133471+0000","last_unstale":"2024-04-25T04:44:24.729471+0000","last_undegraded":"2024-04-25T04:44:24.729471+0000","last_fullsized":"2024-04-25T04:44:24.729471+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_clean_scrub_stamp":"2024-04-25T03:21:53.124515+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:53:23.257337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3835,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.388849+0000","last_change":"2024-04-25T03:22:00.451023+0000","last_active":"2024-04-25T04:44:26.388849+0000","last_peered":"2024-04-25T04:44:26.388849+0000","last_clean":"2024-04-25T04:44:26.388849+0000","last_became_active":"2024-04-25T03:22:00.450704+0000","last_became_peered":"2024-04-25T03:22:00.450704+0000","last_unstale":"2024-04-25T04:44:26.388849+0000","last_undegraded":"2024-04-25T04:44:26.388849+0000","last_fullsized":"2024-04-25T04:44:26.388849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:05.312724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.727707+0000","last_change":"2024-04-25T03:22:00.453301+0000","last_active":"2024-04-25T04:44:24.727707+0000","last_peered":"2024-04-25T04:44:24.727707+0000","last_clean":"2024-04-25T04:44:24.727707+0000","last_became_active":"2024-04-25T03:22:00.453022+0000","last_became_peered":"2024-04-25T03:22:00.453022+0000","last_unstale":"2024-04-25T04:44:24.727707+0000","last_undegraded":"2024-04-25T04:44:24.727707+0000","last_fullsized":"2024-04-25T04:44:24.727707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:38.378533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.726881+0000","last_change":"2024-04-25T03:22:00.454554+0000","last_active":"2024-04-25T04:44:24.726881+0000","last_peered":"2024-04-25T04:44:24.726881+0000","last_clean":"2024-04-25T04:44:24.726881+0000","last_became_active":"2024-04-25T03:22:00.454254+0000","last_became_peered":"2024-04-25T03:22:00.454254+0000","last_unstale":"2024-04-25T04:44:24.726881+0000","last_undegraded":"2024-04-25T04:44:24.726881+0000","last_fullsized":"2024-04-25T04:44:24.726881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:08.518908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729702+0000","last_change":"2024-04-25T03:22:00.454572+0000","last_active":"2024-04-25T04:44:24.729702+0000","last_peered":"2024-04-25T04:44:24.729702+0000","last_clean":"2024-04-25T04:44:24.729702+0000","last_became_active":"2024-04-25T03:22:00.454212+0000","last_became_peered":"2024-04-25T03:22:00.454212+0000","last_unstale":"2024-04-25T04:44:24.729702+0000","last_undegraded":"2024-04-25T04:44:24.729702+0000","last_fullsized":"2024-04-25T04:44:24.729702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:00.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1156540,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":1150142,"kb_avail":93215300,"statfs":{"total":96636764160,"available":95452467200,"internally_reserved":0,"allocated":6537216,"data_stored":3397462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177745861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1377860,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1370878,"kb_avail":92993980,"statfs":{"total":96636764160,"available":95225835520,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1403779525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1265348,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1258366,"kb_avail":93106492,"statfs":{"total":96636764160,"available":95341047808,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1288567237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:45:15.034 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-04-25T04:45:15.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-25T04:45:15.441 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T04:45:15.444 INFO:teuthology.orchestra.run.smithi173.stderr:dumped all 2024-04-25T04:45:15.462 INFO:teuthology.orchestra.run.smithi173.stdout:{"pg_ready":true,"pg_map":{"version":4402,"stamp":"2024-04-25T04:45:14.088751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3799748,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":3779387,"kb_avail":279315772,"statfs":{"total":289910292480,"available":286019350528,"internally_reserved":0,"allocated":20799488,"data_stored":11373122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":3870092623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002420"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.731051+0000","last_change":"2024-04-25T03:22:00.453221+0000","last_active":"2024-04-25T04:44:24.731051+0000","last_peered":"2024-04-25T04:44:24.731051+0000","last_clean":"2024-04-25T04:44:24.731051+0000","last_became_active":"2024-04-25T03:22:00.452902+0000","last_became_peered":"2024-04-25T03:22:00.452902+0000","last_unstale":"2024-04-25T04:44:24.731051+0000","last_undegraded":"2024-04-25T04:44:24.731051+0000","last_fullsized":"2024-04-25T04:44:24.731051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:35:05.745342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3833,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.083180+0000","last_change":"2024-04-25T03:22:00.452610+0000","last_active":"2024-04-25T04:44:26.083180+0000","last_peered":"2024-04-25T04:44:26.083180+0000","last_clean":"2024-04-25T04:44:26.083180+0000","last_became_active":"2024-04-25T03:22:00.452291+0000","last_became_peered":"2024-04-25T03:22:00.452291+0000","last_unstale":"2024-04-25T04:44:26.083180+0000","last_undegraded":"2024-04-25T04:44:26.083180+0000","last_fullsized":"2024-04-25T04:44:26.083180+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:42:10.308241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3833,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.390015+0000","last_change":"2024-04-25T03:22:00.452201+0000","last_active":"2024-04-25T04:44:26.390015+0000","last_peered":"2024-04-25T04:44:26.390015+0000","last_clean":"2024-04-25T04:44:26.390015+0000","last_became_active":"2024-04-25T03:22:00.451642+0000","last_became_peered":"2024-04-25T03:22:00.451642+0000","last_unstale":"2024-04-25T04:44:26.390015+0000","last_undegraded":"2024-04-25T04:44:26.390015+0000","last_fullsized":"2024-04-25T04:44:26.390015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:31.160601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729721+0000","last_change":"2024-04-25T03:22:00.450555+0000","last_active":"2024-04-25T04:44:24.729721+0000","last_peered":"2024-04-25T04:44:24.729721+0000","last_clean":"2024-04-25T04:44:24.729721+0000","last_became_active":"2024-04-25T03:22:00.450263+0000","last_became_peered":"2024-04-25T03:22:00.450263+0000","last_unstale":"2024-04-25T04:44:24.729721+0000","last_undegraded":"2024-04-25T04:44:24.729721+0000","last_fullsized":"2024-04-25T04:44:24.729721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:21:57.478981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":3967,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729471+0000","last_change":"2024-04-25T03:21:54.133861+0000","last_active":"2024-04-25T04:44:24.729471+0000","last_peered":"2024-04-25T04:44:24.729471+0000","last_clean":"2024-04-25T04:44:24.729471+0000","last_became_active":"2024-04-25T03:21:54.133471+0000","last_became_peered":"2024-04-25T03:21:54.133471+0000","last_unstale":"2024-04-25T04:44:24.729471+0000","last_undegraded":"2024-04-25T04:44:24.729471+0000","last_fullsized":"2024-04-25T04:44:24.729471+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_clean_scrub_stamp":"2024-04-25T03:21:53.124515+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:53:23.257337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3835,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.388849+0000","last_change":"2024-04-25T03:22:00.451023+0000","last_active":"2024-04-25T04:44:26.388849+0000","last_peered":"2024-04-25T04:44:26.388849+0000","last_clean":"2024-04-25T04:44:26.388849+0000","last_became_active":"2024-04-25T03:22:00.450704+0000","last_became_peered":"2024-04-25T03:22:00.450704+0000","last_unstale":"2024-04-25T04:44:26.388849+0000","last_undegraded":"2024-04-25T04:44:26.388849+0000","last_fullsized":"2024-04-25T04:44:26.388849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:05.312724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.727707+0000","last_change":"2024-04-25T03:22:00.453301+0000","last_active":"2024-04-25T04:44:24.727707+0000","last_peered":"2024-04-25T04:44:24.727707+0000","last_clean":"2024-04-25T04:44:24.727707+0000","last_became_active":"2024-04-25T03:22:00.453022+0000","last_became_peered":"2024-04-25T03:22:00.453022+0000","last_unstale":"2024-04-25T04:44:24.727707+0000","last_undegraded":"2024-04-25T04:44:24.727707+0000","last_fullsized":"2024-04-25T04:44:24.727707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:38.378533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.726881+0000","last_change":"2024-04-25T03:22:00.454554+0000","last_active":"2024-04-25T04:44:24.726881+0000","last_peered":"2024-04-25T04:44:24.726881+0000","last_clean":"2024-04-25T04:44:24.726881+0000","last_became_active":"2024-04-25T03:22:00.454254+0000","last_became_peered":"2024-04-25T03:22:00.454254+0000","last_unstale":"2024-04-25T04:44:24.726881+0000","last_undegraded":"2024-04-25T04:44:24.726881+0000","last_fullsized":"2024-04-25T04:44:24.726881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:08.518908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729702+0000","last_change":"2024-04-25T03:22:00.454572+0000","last_active":"2024-04-25T04:44:24.729702+0000","last_peered":"2024-04-25T04:44:24.729702+0000","last_clean":"2024-04-25T04:44:24.729702+0000","last_became_active":"2024-04-25T03:22:00.454212+0000","last_became_peered":"2024-04-25T03:22:00.454212+0000","last_unstale":"2024-04-25T04:44:24.729702+0000","last_undegraded":"2024-04-25T04:44:24.729702+0000","last_fullsized":"2024-04-25T04:44:24.729702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:00.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1156540,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":1150142,"kb_avail":93215300,"statfs":{"total":96636764160,"available":95452467200,"internally_reserved":0,"allocated":6537216,"data_stored":3397462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177745861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1377860,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1370878,"kb_avail":92993980,"statfs":{"total":96636764160,"available":95225835520,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1403779525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1265348,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1258366,"kb_avail":93106492,"statfs":{"total":96636764160,"available":95341047808,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1288567237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:45:15.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-04-25T04:45:15.876 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T04:45:15.876 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1906,"fsid":"2d0af0d3-7b88-4ef9-a498-864f18964d8a","created":"2024-04-25T03:21:42.734143+0000","modified":"2024-04-25T04:44:24.720602+0000","last_up_change":"2024-04-25T03:21:52.123612+0000","last_in_change":"2024-04-25T03:21:44.102108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T03:21:52.820161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-25T03:21:58.844332+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83fb82c9-be2c-46f4-962d-3d04a08913bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6805","nonce":38674190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6810","nonce":38674190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6815","nonce":38674190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":38674190},{"type":"v1","addr":"172.21.15.173:6813","nonce":38674190}]},"public_addr":"172.21.15.173:6805/38674190","cluster_addr":"172.21.15.173:6810/38674190","heartbeat_back_addr":"172.21.15.173:6815/38674190","heartbeat_front_addr":"172.21.15.173:6813/38674190","state":["exists","up"]},{"osd":1,"uuid":"adefaee8-cbd6-47a4-abdc-55ff3a06d718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6801","nonce":2340622026}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6804","nonce":2340622026}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6811","nonce":2340622026}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2340622026},{"type":"v1","addr":"172.21.15.173:6808","nonce":2340622026}]},"public_addr":"172.21.15.173:6801/2340622026","cluster_addr":"172.21.15.173:6804/2340622026","heartbeat_back_addr":"172.21.15.173:6811/2340622026","heartbeat_front_addr":"172.21.15.173:6808/2340622026","state":["exists","up"]},{"osd":2,"uuid":"3b5c6e5b-87f8-40be-94f7-ea31ebb1bf8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6817","nonce":193268527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6819","nonce":193268527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6823","nonce":193268527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":193268527},{"type":"v1","addr":"172.21.15.173:6821","nonce":193268527}]},"public_addr":"172.21.15.173:6817/193268527","cluster_addr":"172.21.15.173:6819/193268527","heartbeat_back_addr":"172.21.15.173:6823/193268527","heartbeat_front_addr":"172.21.15.173:6821/193268527","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.716608+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:49.500194+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T03:21:51.122866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2979,"length":2},{"begin":2989,"length":1},{"begin":3002,"length":1},{"begin":3272,"length":1},{"begin":3274,"length":1},{"begin":3289,"length":1},{"begin":3293,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T04:45:16.897 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-04-25T04:45:16.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T04:45:17.126 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T04:45:17.126 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T04:45:17.126 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T04:45:17.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-04-25T04:45:17.555 INFO:teuthology.orchestra.run.smithi173.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T04:45:17.572 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-04-25T04:45:17.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T04:45:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T04:45:17.796 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T04:45:17.796 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T04:45:17.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-04-25T04:45:18.246 INFO:teuthology.orchestra.run.smithi173.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T04:45:18.263 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-04-25T04:45:18.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T04:45:18.487 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T04:45:18.487 INFO:teuthology.orchestra.run.smithi173.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T04:45:18.487 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T04:45:18.505 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-04-25T04:45:18.920 INFO:teuthology.orchestra.run.smithi173.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T04:45:18.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-25T04:45:19.349 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T04:45:19.352 INFO:teuthology.orchestra.run.smithi173.stderr:dumped all 2024-04-25T04:45:19.370 INFO:teuthology.orchestra.run.smithi173.stdout:{"pg_ready":true,"pg_map":{"version":4404,"stamp":"2024-04-25T04:45:18.089693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3799748,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":3779387,"kb_avail":279315772,"statfs":{"total":289910292480,"available":286019350528,"internally_reserved":0,"allocated":20799488,"data_stored":11373122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":3870092623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002605"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.731051+0000","last_change":"2024-04-25T03:22:00.453221+0000","last_active":"2024-04-25T04:44:24.731051+0000","last_peered":"2024-04-25T04:44:24.731051+0000","last_clean":"2024-04-25T04:44:24.731051+0000","last_became_active":"2024-04-25T03:22:00.452902+0000","last_became_peered":"2024-04-25T03:22:00.452902+0000","last_unstale":"2024-04-25T04:44:24.731051+0000","last_undegraded":"2024-04-25T04:44:24.731051+0000","last_fullsized":"2024-04-25T04:44:24.731051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:35:05.745342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3842,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:18.043478+0000","last_change":"2024-04-25T04:45:18.043501+0000","last_active":"2024-04-25T04:45:18.043478+0000","last_peered":"2024-04-25T04:45:18.043478+0000","last_clean":"2024-04-25T04:45:18.043478+0000","last_became_active":"2024-04-25T03:22:00.452291+0000","last_became_peered":"2024-04-25T03:22:00.452291+0000","last_unstale":"2024-04-25T04:45:18.043478+0000","last_undegraded":"2024-04-25T04:45:18.043478+0000","last_fullsized":"2024-04-25T04:45:18.043478+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:18.043375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:18.043375+0000","last_clean_scrub_stamp":"2024-04-25T04:45:18.043375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:39.986728+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3833,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.390015+0000","last_change":"2024-04-25T03:22:00.452201+0000","last_active":"2024-04-25T04:44:26.390015+0000","last_peered":"2024-04-25T04:44:26.390015+0000","last_clean":"2024-04-25T04:44:26.390015+0000","last_became_active":"2024-04-25T03:22:00.451642+0000","last_became_peered":"2024-04-25T03:22:00.451642+0000","last_unstale":"2024-04-25T04:44:26.390015+0000","last_undegraded":"2024-04-25T04:44:26.390015+0000","last_fullsized":"2024-04-25T04:44:26.390015+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:31.160601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729721+0000","last_change":"2024-04-25T03:22:00.450555+0000","last_active":"2024-04-25T04:44:24.729721+0000","last_peered":"2024-04-25T04:44:24.729721+0000","last_clean":"2024-04-25T04:44:24.729721+0000","last_became_active":"2024-04-25T03:22:00.450263+0000","last_became_peered":"2024-04-25T03:22:00.450263+0000","last_unstale":"2024-04-25T04:44:24.729721+0000","last_undegraded":"2024-04-25T04:44:24.729721+0000","last_fullsized":"2024-04-25T04:44:24.729721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:21:57.478981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":3967,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729471+0000","last_change":"2024-04-25T03:21:54.133861+0000","last_active":"2024-04-25T04:44:24.729471+0000","last_peered":"2024-04-25T04:44:24.729471+0000","last_clean":"2024-04-25T04:44:24.729471+0000","last_became_active":"2024-04-25T03:21:54.133471+0000","last_became_peered":"2024-04-25T03:21:54.133471+0000","last_unstale":"2024-04-25T04:44:24.729471+0000","last_undegraded":"2024-04-25T04:44:24.729471+0000","last_fullsized":"2024-04-25T04:44:24.729471+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:53.124515+0000","last_clean_scrub_stamp":"2024-04-25T03:21:53.124515+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:53:23.257337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3835,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:26.388849+0000","last_change":"2024-04-25T03:22:00.451023+0000","last_active":"2024-04-25T04:44:26.388849+0000","last_peered":"2024-04-25T04:44:26.388849+0000","last_clean":"2024-04-25T04:44:26.388849+0000","last_became_active":"2024-04-25T03:22:00.450704+0000","last_became_peered":"2024-04-25T03:22:00.450704+0000","last_unstale":"2024-04-25T04:44:26.388849+0000","last_undegraded":"2024-04-25T04:44:26.388849+0000","last_fullsized":"2024-04-25T04:44:26.388849+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:16:05.312724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3832,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.727707+0000","last_change":"2024-04-25T03:22:00.453301+0000","last_active":"2024-04-25T04:44:24.727707+0000","last_peered":"2024-04-25T04:44:24.727707+0000","last_clean":"2024-04-25T04:44:24.727707+0000","last_became_active":"2024-04-25T03:22:00.453022+0000","last_became_peered":"2024-04-25T03:22:00.453022+0000","last_unstale":"2024-04-25T04:44:24.727707+0000","last_undegraded":"2024-04-25T04:44:24.727707+0000","last_fullsized":"2024-04-25T04:44:24.727707+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:35:38.378533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.726881+0000","last_change":"2024-04-25T03:22:00.454554+0000","last_active":"2024-04-25T04:44:24.726881+0000","last_peered":"2024-04-25T04:44:24.726881+0000","last_clean":"2024-04-25T04:44:24.726881+0000","last_became_active":"2024-04-25T03:22:00.454254+0000","last_became_peered":"2024-04-25T03:22:00.454254+0000","last_unstale":"2024-04-25T04:44:24.726881+0000","last_undegraded":"2024-04-25T04:44:24.726881+0000","last_fullsized":"2024-04-25T04:44:24.726881+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:03:08.518908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3831,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:44:24.729702+0000","last_change":"2024-04-25T03:22:00.454572+0000","last_active":"2024-04-25T04:44:24.729702+0000","last_peered":"2024-04-25T04:44:24.729702+0000","last_clean":"2024-04-25T04:44:24.729702+0000","last_became_active":"2024-04-25T03:22:00.454212+0000","last_became_peered":"2024-04-25T03:22:00.454212+0000","last_unstale":"2024-04-25T04:44:24.729702+0000","last_undegraded":"2024-04-25T04:44:24.729702+0000","last_fullsized":"2024-04-25T04:44:24.729702+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T03:21:59.434205+0000","last_clean_scrub_stamp":"2024-04-25T03:21:59.434205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:00.914785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706666,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1156540,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":1150142,"kb_avail":93215300,"statfs":{"total":96636764160,"available":95452467200,"internally_reserved":0,"allocated":6537216,"data_stored":3397462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177745861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1377860,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1370878,"kb_avail":92993980,"statfs":{"total":96636764160,"available":95225835520,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1403779525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706667,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1265348,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1258366,"kb_avail":93106492,"statfs":{"total":96636764160,"available":95341047808,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1288567237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:45:19.371 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-25T03:21:59.434205+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=59, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.371 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-25T03:21:59.434205+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=59, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.371 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T03:21:59.434205+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=59, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.371 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T03:21:53.124515+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=53, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.371 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T03:21:59.434205+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=59, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.371 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T03:21:59.434205+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=59, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.371 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-25T03:21:59.434205+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=59, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.372 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T03:21:59.434205+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=3, tm_min=21, tm_sec=59, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=4, tm_min=45, tm_sec=15, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T04:45:19.372 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T04:45:39.373 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-04-25T04:45:39.792 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T04:45:39.795 INFO:teuthology.orchestra.run.smithi173.stderr:dumped all 2024-04-25T04:45:39.813 INFO:teuthology.orchestra.run.smithi173.stdout:{"pg_ready":true,"pg_map":{"version":4414,"stamp":"2024-04-25T04:45:38.094061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3799728,"kb_used_data":20312,"kb_used_omap":4,"kb_used_meta":3779387,"kb_avail":279315792,"statfs":{"total":289910292480,"available":286019371008,"internally_reserved":0,"allocated":20799488,"data_stored":11373122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":3870092623},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002592"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":3842,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:20.819427+0000","last_change":"2024-04-25T04:45:20.819427+0000","last_active":"2024-04-25T04:45:20.819427+0000","last_peered":"2024-04-25T04:45:20.819427+0000","last_clean":"2024-04-25T04:45:20.819427+0000","last_became_active":"2024-04-25T03:22:00.452902+0000","last_became_peered":"2024-04-25T03:22:00.452902+0000","last_unstale":"2024-04-25T04:45:20.819427+0000","last_undegraded":"2024-04-25T04:45:20.819427+0000","last_fullsized":"2024-04-25T04:45:20.819427+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:20.819311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:20.819311+0000","last_clean_scrub_stamp":"2024-04-25T04:45:20.819311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:50:23.301604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3843,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:18.043501+0000","last_change":"2024-04-25T04:45:18.043501+0000","last_active":"2024-04-25T04:45:18.043501+0000","last_peered":"2024-04-25T04:45:18.043501+0000","last_clean":"2024-04-25T04:45:18.043501+0000","last_became_active":"2024-04-25T03:22:00.452291+0000","last_became_peered":"2024-04-25T03:22:00.452291+0000","last_unstale":"2024-04-25T04:45:18.043501+0000","last_undegraded":"2024-04-25T04:45:18.043501+0000","last_fullsized":"2024-04-25T04:45:18.043501+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:18.043375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:18.043375+0000","last_clean_scrub_stamp":"2024-04-25T04:45:18.043375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:39.986728+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3843,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:20.345827+0000","last_change":"2024-04-25T04:45:20.345827+0000","last_active":"2024-04-25T04:45:20.345827+0000","last_peered":"2024-04-25T04:45:20.345827+0000","last_clean":"2024-04-25T04:45:20.345827+0000","last_became_active":"2024-04-25T03:22:00.451642+0000","last_became_peered":"2024-04-25T03:22:00.451642+0000","last_unstale":"2024-04-25T04:45:20.345827+0000","last_undegraded":"2024-04-25T04:45:20.345827+0000","last_fullsized":"2024-04-25T04:45:20.345827+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:20.345665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:20.345665+0000","last_clean_scrub_stamp":"2024-04-25T04:45:20.345665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:41:00.955497+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":3842,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:19.935629+0000","last_change":"2024-04-25T04:45:19.935629+0000","last_active":"2024-04-25T04:45:19.935629+0000","last_peered":"2024-04-25T04:45:19.935629+0000","last_clean":"2024-04-25T04:45:19.935629+0000","last_became_active":"2024-04-25T03:22:00.450263+0000","last_became_peered":"2024-04-25T03:22:00.450263+0000","last_unstale":"2024-04-25T04:45:19.935629+0000","last_undegraded":"2024-04-25T04:45:19.935629+0000","last_fullsized":"2024-04-25T04:45:19.935629+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:19.935501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:19.935501+0000","last_clean_scrub_stamp":"2024-04-25T04:45:19.935501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:37:07.126255+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1906'99","reported_seq":3978,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:18.825080+0000","last_change":"2024-04-25T04:45:18.825108+0000","last_active":"2024-04-25T04:45:18.825080+0000","last_peered":"2024-04-25T04:45:18.825080+0000","last_clean":"2024-04-25T04:45:18.825080+0000","last_became_active":"2024-04-25T03:21:54.133471+0000","last_became_peered":"2024-04-25T03:21:54.133471+0000","last_unstale":"2024-04-25T04:45:18.825080+0000","last_undegraded":"2024-04-25T04:45:18.825080+0000","last_fullsized":"2024-04-25T04:45:18.825080+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"1906'99","last_scrub_stamp":"2024-04-25T04:45:18.825002+0000","last_deep_scrub":"1906'99","last_deep_scrub_stamp":"2024-04-25T04:45:18.825002+0000","last_clean_scrub_stamp":"2024-04-25T04:45:18.825002+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:56.481192+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3844,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:19.296427+0000","last_change":"2024-04-25T04:45:19.296449+0000","last_active":"2024-04-25T04:45:19.296427+0000","last_peered":"2024-04-25T04:45:19.296427+0000","last_clean":"2024-04-25T04:45:19.296427+0000","last_became_active":"2024-04-25T03:22:00.450704+0000","last_became_peered":"2024-04-25T03:22:00.450704+0000","last_unstale":"2024-04-25T04:45:19.296427+0000","last_undegraded":"2024-04-25T04:45:19.296427+0000","last_fullsized":"2024-04-25T04:45:19.296427+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:19.296317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:19.296317+0000","last_clean_scrub_stamp":"2024-04-25T04:45:19.296317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:20:35.809831+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3842,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:21.950588+0000","last_change":"2024-04-25T04:45:21.950588+0000","last_active":"2024-04-25T04:45:21.950588+0000","last_peered":"2024-04-25T04:45:21.950588+0000","last_clean":"2024-04-25T04:45:21.950588+0000","last_became_active":"2024-04-25T03:22:00.453022+0000","last_became_peered":"2024-04-25T03:22:00.453022+0000","last_unstale":"2024-04-25T04:45:21.950588+0000","last_undegraded":"2024-04-25T04:45:21.950588+0000","last_fullsized":"2024-04-25T04:45:21.950588+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:21.950427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:21.950427+0000","last_clean_scrub_stamp":"2024-04-25T04:45:21.950427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:22:52.807387+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3840,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:22.845487+0000","last_change":"2024-04-25T04:45:22.845512+0000","last_active":"2024-04-25T04:45:22.845487+0000","last_peered":"2024-04-25T04:45:22.845487+0000","last_clean":"2024-04-25T04:45:22.845487+0000","last_became_active":"2024-04-25T03:22:00.454254+0000","last_became_peered":"2024-04-25T03:22:00.454254+0000","last_unstale":"2024-04-25T04:45:22.845487+0000","last_undegraded":"2024-04-25T04:45:22.845487+0000","last_fullsized":"2024-04-25T04:45:22.845487+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:22.845322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:22.845322+0000","last_clean_scrub_stamp":"2024-04-25T04:45:22.845322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:33:59.553457+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":3840,"reported_epoch":1906,"state":"active+clean","last_fresh":"2024-04-25T04:45:23.953214+0000","last_change":"2024-04-25T04:45:23.953236+0000","last_active":"2024-04-25T04:45:23.953214+0000","last_peered":"2024-04-25T04:45:23.953214+0000","last_clean":"2024-04-25T04:45:23.953214+0000","last_became_active":"2024-04-25T03:22:00.454212+0000","last_became_peered":"2024-04-25T03:22:00.454212+0000","last_unstale":"2024-04-25T04:45:23.953214+0000","last_undegraded":"2024-04-25T04:45:23.953214+0000","last_fullsized":"2024-04-25T04:45:23.953214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T04:45:23.953099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T04:45:23.953099+0000","last_clean_scrub_stamp":"2024-04-25T04:45:23.953099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T13:03:34.138120+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1156536,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":1150142,"kb_avail":93215304,"statfs":{"total":96636764160,"available":95452471296,"internally_reserved":0,"allocated":6537216,"data_stored":3397462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1177745861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1377852,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1370878,"kb_avail":92993988,"statfs":{"total":96636764160,"available":95225843712,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1403779525},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654706671,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1265340,"kb_used_data":6964,"kb_used_omap":1,"kb_used_meta":1258366,"kb_avail":93106500,"statfs":{"total":96636764160,"available":95341056000,"internally_reserved":0,"allocated":7131136,"data_stored":3987830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1288567237},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T04:45:39.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-04-25T04:45:40.247 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T04:45:40.248 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T04:45:40.248 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-04-25T04:45:40.248 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:40.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-04-25T04:45:46.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~5s 2024-04-25T04:45:46.349 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-04-25T04:45:46.350 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-04-25T04:45:46.350 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:51.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-04-25T04:45:51.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-04-25T04:45:52.452 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-04-25T04:45:52.452 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-04-25T04:45:52.453 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:56.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-04-25T04:45:56.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-04-25T04:45:58.555 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-04-25T04:45:58.555 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T04:45:58.555 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-04-25T04:45:58.555 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:58.631 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-04-25T04:45:58.631 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T04:45:58.632 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-04-25T04:45:58.632 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T04:45:58.667 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-04-25T04:45:58.667 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T04:45:58.668 DEBUG:teuthology.orchestra.run.smithi173:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-25T04:45:58.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T04:45:58.832 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-04-25T04:45:59.128 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T04:45:59.129 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-04-25T04:45:59.366 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T04:45:59.366 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-04-25T04:45:59.622 INFO:tasks.ceph:Archiving mon data... 2024-04-25T04:45:59.622 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/data/cluster1.mon.a.tgz 2024-04-25T04:45:59.622 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-04-25T04:45:59.671 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.STD2QACenY 2024-04-25T04:45:59.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.STD2QACenY 2024-04-25T04:45:59.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.STD2QACenY 2024-04-25T04:46:00.065 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.STD2QACenY is 528KB 2024-04-25T04:46:00.167 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.STD2QACenY 2024-04-25T04:46:00.212 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T04:46:00.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T04:46:00.256 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T04:46:00.439 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T04:46:00.504 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T04:46:00.504 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T04:46:00.506 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/remote/smithi173/crash 2024-04-25T04:46:00.506 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T04:46:00.572 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/remote/smithi188/crash 2024-04-25T04:46:00.573 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T04:46:00.641 INFO:tasks.ceph:Compressing logs... 2024-04-25T04:46:00.642 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T04:46:00.646 DEBUG:teuthology.orchestra.run.smithi188:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T04:46:00.722 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81023.log 2024-04-25T04:46:00.723 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117396.log 2024-04-25T04:46:00.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81023.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81023.log.gz 2024-04-25T04:46:00.724 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54031.log 2024-04-25T04:46:00.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117396.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117396.log.gz 2024-04-25T04:46:00.724 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60949.log 2024-04-25T04:46:00.725 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134023.log 2024-04-25T04:46:00.726 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107329.log 2024-04-25T04:46:00.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134023.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134023.log.gz 2024-04-25T04:46:00.727 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70112.log 2024-04-25T04:46:00.728 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107329.log.gz 2024-04-25T04:46:00.728 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66387.log 2024-04-25T04:46:00.729 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70112.log.gz 2024-04-25T04:46:00.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148813.log 2024-04-25T04:46:00.729 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66387.log.gz 2024-04-25T04:46:00.730 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117253.log 2024-04-25T04:46:00.730 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148813.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148813.log.gz 2024-04-25T04:46:00.730 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116107.log 2024-04-25T04:46:00.731 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117253.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117253.log.gz 2024-04-25T04:46:00.731 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107153.log 2024-04-25T04:46:00.732 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116107.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116107.log.gz 2024-04-25T04:46:00.732 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59615.log 2024-04-25T04:46:00.733 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107153.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54031.log.gz 2024-04-25T04:46:00.733 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107153.log.gz 2024-04-25T04:46:00.733 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60949.log.gz 2024-04-25T04:46:00.734 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101214.log 2024-04-25T04:46:00.734 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59615.log.gz 2024-04-25T04:46:00.734 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150446.log 2024-04-25T04:46:00.735 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101214.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101214.log.gz 2024-04-25T04:46:00.735 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154768.log 2024-04-25T04:46:00.736 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45424.log 2024-04-25T04:46:00.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150446.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150446.log.gz 2024-04-25T04:46:00.737 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71308.log 2024-04-25T04:46:00.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154768.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154768.log.gz 2024-04-25T04:46:00.737 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91833.log 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39893.log 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45424.log.gz 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46010.log 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71308.log.gz 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95521.log 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91833.log.gz 2024-04-25T04:46:00.739 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143074.log 2024-04-25T04:46:00.742 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.39893.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39893.log.gz 2024-04-25T04:46:00.742 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40636.log 2024-04-25T04:46:00.742 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.46010.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46010.log.gz 2024-04-25T04:46:00.742 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40343.log 2024-04-25T04:46:00.742 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40636.log.gz 2024-04-25T04:46:00.742 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40196.log 2024-04-25T04:46:00.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95521.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95521.log.gz 2024-04-25T04:46:00.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55679.log 2024-04-25T04:46:00.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143074.log.gz 2024-04-25T04:46:00.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61556.log 2024-04-25T04:46:00.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55679.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55679.log.gz 2024-04-25T04:46:00.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127329.log 2024-04-25T04:46:00.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61556.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61556.log.gz 2024-04-25T04:46:00.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94041.log 2024-04-25T04:46:00.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127329.log.gz 2024-04-25T04:46:00.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118491.log 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94041.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94041.log.gz 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126501.log 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118491.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118491.log.gz 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155805.log 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126501.log.gz 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72705.log 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155805.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155805.log.gz 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41193.log 2024-04-25T04:46:00.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72705.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72705.log.gz 2024-04-25T04:46:00.750 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117018.log 2024-04-25T04:46:00.750 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41193.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41193.log.gz 2024-04-25T04:46:00.750 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108604.log 2024-04-25T04:46:00.751 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40343.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40343.log.gz 2024-04-25T04:46:00.751 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39733.log 2024-04-25T04:46:00.751 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40196.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40196.log.gz 2024-04-25T04:46:00.751 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-04-25T04:46:00.751 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.39733.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39733.log.gz 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45767.log 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45865.log 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45767.log.gz 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39942.log 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45865.log: 0.0%gzip -5 -- replaced with /var/log/ceph/cluster1-client.admin.45865.log.gz--verbose -- 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40481.log 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.39942.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39942.log.gz 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45669.log 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40481.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40481.log.gz 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40097.log 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45669.log.gz 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45816.log 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36077.log 0.0% 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.40097.log.gz 2024-04-25T04:46:00.752 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45816.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45816.log.gz 2024-04-25T04:46:00.753 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39991.log 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117018.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117018.log.gz 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135942.log 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108604.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108604.log.gz 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70462.log 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135942.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135942.log.gz 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69632.log 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70462.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70462.log.gz 2024-04-25T04:46:00.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52103.log 2024-04-25T04:46:00.755 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69632.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69632.log.gz 2024-04-25T04:46:00.755 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.102853.log 2024-04-25T04:46:00.755 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52103.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52103.log.gz 2024-04-25T04:46:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57085.log 2024-04-25T04:46:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.102853.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.102853.log.gz 2024-04-25T04:46:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53391.log 2024-04-25T04:46:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57085.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57085.log.gz 2024-04-25T04:46:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152992.log 2024-04-25T04:46:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53391.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53391.log.gz 2024-04-25T04:46:00.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56143.log 2024-04-25T04:46:00.761 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.36077.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36077.log.gz 2024-04-25T04:46:00.761 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36174.log 2024-04-25T04:46:00.761 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.39991.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39991.log.gz 2024-04-25T04:46:00.761 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40198.log 2024-04-25T04:46:00.761 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.36174.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36174.log.gz 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45914.log 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40198.log.gz 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40344.log 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45914.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45914.log.gz 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35884.log 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40344.log.gz 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40048.log 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.35884.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35884.log.gz 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40345.log 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40048.log.gz 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45620.log 2024-04-25T04:46:00.762 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40345.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40345.log.gz 2024-04-25T04:46:00.763 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45571.log 2024-04-25T04:46:00.763 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45620.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45620.log.gz 2024-04-25T04:46:00.763 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-04-25T04:46:00.765 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45571.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45571.log.gz 2024-04-25T04:46:00.766 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36125.log 2024-04-25T04:46:00.766 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36372.log 2024-04-25T04:46:00.766 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.36125.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36125.log.gz 2024-04-25T04:46:00.766 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-04-25T04:46:00.766 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.36372.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36372.log.gz 2024-04-25T04:46:00.766 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31303.log 2024-04-25T04:46:00.773 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39824.log 2024-04-25T04:46:00.773 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/ceph.tmp-client.admin.31303.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31303.log.gz 2024-04-25T04:46:00.773 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40685.log 2024-04-25T04:46:00.773 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.39824.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39824.log.gz 2024-04-25T04:46:00.773 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40587.log 2024-04-25T04:46:00.773 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40685.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40685.log.gz 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45718.log 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40587.log.gz 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40492.log 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45718.log.gz 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45522.log 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40492.log.gz 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr: 87.5% -- replaced with /var/log/ceph/cluster1.log.gz 2024-04-25T04:46:00.774 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-04-25T04:46:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45522.log.gz 2024-04-25T04:46:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-04-25T04:46:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40482.log 2024-04-25T04:46:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45963.log 2024-04-25T04:46:00.782 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40482.log: gzip -5 0.0%--verbose -- /var/log/ceph/cluster1-client.admin.40197.log -- replaced with /var/log/ceph/cluster1-client.admin.40482.log.gz 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45963.log.gz 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.40197.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40197.log.gz 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36300.log 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45473.log 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.36300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36300.log.gz 2024-04-25T04:46:00.783 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cluster1-client.admin.45473.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45473.log.gz 2024-04-25T04:46:00.791 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152992.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152992.log.gz 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108168.log 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56143.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56143.log.gz 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150938.log 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108168.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108168.log.gz 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114602.log 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150938.log.gz 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118425.log 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.114602.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114602.log.gz 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156653.log 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118425.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118425.log.gz 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82195.log 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156653.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156653.log.gz 2024-04-25T04:46:00.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144992.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82195.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82195.log.gz 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91118.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144992.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144992.log.gz 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41892.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91118.log.gz 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74975.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41892.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41892.log.gz 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87218.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74975.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74975.log.gz 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53803.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87218.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87218.log.gz 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153681.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53803.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53803.log.gz 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81312.log 2024-04-25T04:46:00.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153681.log.gz 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84772.log 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81312.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81312.log.gz 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147269.log 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84772.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84772.log.gz 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93461.log 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147269.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147269.log.gz 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67523.log 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93461.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93461.log.gz 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92991.log 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67523.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67523.log.gz 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150973.log 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92991.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92991.log.gz 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65157.log 2024-04-25T04:46:00.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150973.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150973.log.gz 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51326.log 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65157.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65157.log.gz 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137878.log 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51326.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51326.log.gz 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82932.log 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137878.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137878.log.gz 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130024.log 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82932.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82932.log.gz 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69227.log 2024-04-25T04:46:00.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130024.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130024.log.gz 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160128.log 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69227.log.gz 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110550.log 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160128.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160128.log.gz 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61302.log 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110550.log.gz 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155438.log 2024-04-25T04:46:00.796 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61302.log.gz 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81422.log 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155438.log.gz 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64677.log 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81422.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81422.log.gz 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55568.log 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.64677.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64677.log.gz 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106270.log 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.55568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55568.log.gz 2024-04-25T04:46:00.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143914.log 2024-04-25T04:46:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106270.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106270.log.gz 2024-04-25T04:46:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56291.log 2024-04-25T04:46:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143914.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143914.log.gz 2024-04-25T04:46:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93919.log 2024-04-25T04:46:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56291.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56291.log.gz 2024-04-25T04:46:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90675.log 2024-04-25T04:46:00.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93919.log.gz 2024-04-25T04:46:00.810 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65139.log 2024-04-25T04:46:00.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.90675.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90675.log.gz 2024-04-25T04:46:00.810 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60504.log 2024-04-25T04:46:00.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65139.log.gz 2024-04-25T04:46:00.810 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81421.log 2024-04-25T04:46:00.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60504.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60504.log.gz 2024-04-25T04:46:00.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140856.log 2024-04-25T04:46:00.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81421.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81421.log.gz 2024-04-25T04:46:00.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155710.log 2024-04-25T04:46:00.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140856.log.gz 2024-04-25T04:46:00.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108045.log 2024-04-25T04:46:00.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155710.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155710.log.gz 2024-04-25T04:46:00.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63293.log 2024-04-25T04:46:00.812 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108045.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108045.log.gz 2024-04-25T04:46:00.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45856.log 2024-04-25T04:46:00.812 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63293.log.gz 2024-04-25T04:46:00.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143283.log 2024-04-25T04:46:00.812 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45856.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64034.log 2024-04-25T04:46:00.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143283.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143283.log.gz 2024-04-25T04:46:00.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141233.log 2024-04-25T04:46:00.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.64034.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64034.log.gz 2024-04-25T04:46:00.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158366.log 2024-04-25T04:46:00.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141233.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141233.log.gz 2024-04-25T04:46:00.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114827.log 2024-04-25T04:46:00.814 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.158366.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158366.log.gz 2024-04-25T04:46:00.814 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101670.log 2024-04-25T04:46:00.814 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114827.log.gz 2024-04-25T04:46:00.814 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117815.log 2024-04-25T04:46:00.814 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101670.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101670.log.gz 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156486.log 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117815.log.gz 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58712.log 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156486.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156486.log.gz 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65986.log 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58712.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45856.log.gz 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58712.log.gz 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89542.log 2024-04-25T04:46:00.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65986.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65986.log.gz 2024-04-25T04:46:00.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161264.log 2024-04-25T04:46:00.816 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89542.log.gz 2024-04-25T04:46:00.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120956.log 2024-04-25T04:46:00.816 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161264.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161264.log.gz 2024-04-25T04:46:00.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154682.log 2024-04-25T04:46:00.823 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120956.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120956.log.gz 2024-04-25T04:46:00.823 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131002.log 2024-04-25T04:46:00.823 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154682.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154682.log.gz 2024-04-25T04:46:00.823 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106103.log 2024-04-25T04:46:00.823 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131002.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131002.log.gz 2024-04-25T04:46:00.823 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106103.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106103.log.gz 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147954.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110802.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147954.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147954.log.gz 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137372.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110802.log.gz 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49951.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137372.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137372.log.gz 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133100.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.49951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156576.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49951.log.gz 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133100.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133100.log.gz 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152786.log 2024-04-25T04:46:00.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156576.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156576.log.gz 2024-04-25T04:46:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61514.log 2024-04-25T04:46:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152786.log.gz 2024-04-25T04:46:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54503.log 2024-04-25T04:46:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61514.log.gz 2024-04-25T04:46:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82994.log 2024-04-25T04:46:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54503.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54503.log.gz 2024-04-25T04:46:00.825 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152721.log 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82994.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82994.log.gz 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102243.log 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152721.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152721.log.gz 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46718.log 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102243.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102243.log.gz 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154725.log 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46718.log.gz 2024-04-25T04:46:00.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107330.log 2024-04-25T04:46:00.828 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154725.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154725.log.gz 2024-04-25T04:46:00.829 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43009.log 2024-04-25T04:46:00.829 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107330.log.gz 2024-04-25T04:46:00.829 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154392.log 2024-04-25T04:46:00.829 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.43009.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43009.log.gz 2024-04-25T04:46:00.830 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79379.log 2024-04-25T04:46:00.830 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154392.log.gz 2024-04-25T04:46:00.830 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69138.log 2024-04-25T04:46:00.831 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79379.log.gz 2024-04-25T04:46:00.831 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66658.log 2024-04-25T04:46:00.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69138.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69138.log.gz 2024-04-25T04:46:00.832 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93995.log 2024-04-25T04:46:00.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66658.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66658.log.gz 2024-04-25T04:46:00.833 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120749.log 2024-04-25T04:46:00.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93995.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93995.log.gz 2024-04-25T04:46:00.834 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47225.log 2024-04-25T04:46:00.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120749.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120749.log.gz 2024-04-25T04:46:00.835 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87976.log 2024-04-25T04:46:00.835 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47225.log.gz 2024-04-25T04:46:00.836 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115754.log 2024-04-25T04:46:00.836 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87976.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87976.log.gz 2024-04-25T04:46:00.836 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52540.log 2024-04-25T04:46:00.837 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115754.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115754.log.gz 2024-04-25T04:46:00.837 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118783.log 2024-04-25T04:46:00.838 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52540.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52540.log.gz 2024-04-25T04:46:00.838 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156649.log 2024-04-25T04:46:00.839 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118783.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118783.log.gz 2024-04-25T04:46:00.839 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147362.log 2024-04-25T04:46:00.840 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156649.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156649.log.gz 2024-04-25T04:46:00.840 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149423.log 2024-04-25T04:46:00.841 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147362.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147362.log.gz 2024-04-25T04:46:00.841 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131860.log 2024-04-25T04:46:00.842 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149423.log.gz 2024-04-25T04:46:00.842 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140163.log 2024-04-25T04:46:00.843 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131860.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131860.log.gz 2024-04-25T04:46:00.843 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60687.log 2024-04-25T04:46:00.843 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140163.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140163.log.gz 2024-04-25T04:46:00.844 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146278.log 2024-04-25T04:46:00.844 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60687.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60687.log.gz 2024-04-25T04:46:00.844 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102433.log 2024-04-25T04:46:00.845 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146278.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146278.log.gz 2024-04-25T04:46:00.845 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154767.log 2024-04-25T04:46:00.846 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102433.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102433.log.gz 2024-04-25T04:46:00.846 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81527.log 2024-04-25T04:46:00.847 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154767.log.gz 2024-04-25T04:46:00.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68990.log 2024-04-25T04:46:00.848 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81527.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81527.log.gz 2024-04-25T04:46:00.848 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134046.log 2024-04-25T04:46:00.848 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68990.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68990.log.gz 2024-04-25T04:46:00.849 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97705.log 2024-04-25T04:46:00.849 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134046.log.gz 2024-04-25T04:46:00.850 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57554.log 2024-04-25T04:46:00.851 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97705.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97705.log.gz 2024-04-25T04:46:00.851 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125120.log 2024-04-25T04:46:00.851 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57554.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57554.log.gz 2024-04-25T04:46:00.852 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145522.log 2024-04-25T04:46:00.852 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125120.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125120.log.gz 2024-04-25T04:46:00.852 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78209.log 2024-04-25T04:46:00.853 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145522.log.gz 2024-04-25T04:46:00.853 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44777.log 2024-04-25T04:46:00.854 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78209.log.gz 2024-04-25T04:46:00.854 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106356.log 2024-04-25T04:46:00.855 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44777.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44777.log.gz 2024-04-25T04:46:00.855 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88327.log 2024-04-25T04:46:00.856 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106356.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106356.log.gz 2024-04-25T04:46:00.856 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91210.log 2024-04-25T04:46:00.857 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88327.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88327.log.gz 2024-04-25T04:46:00.857 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105428.log 2024-04-25T04:46:00.858 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91210.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91210.log.gz 2024-04-25T04:46:00.858 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102395.log 2024-04-25T04:46:00.859 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105428.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105428.log.gz 2024-04-25T04:46:00.859 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69118.log 2024-04-25T04:46:00.859 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102395.log.gz 2024-04-25T04:46:00.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120873.log 2024-04-25T04:46:00.860 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69118.log.gz 2024-04-25T04:46:00.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111037.log 2024-04-25T04:46:00.861 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120873.log.gz 2024-04-25T04:46:00.862 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116680.log 2024-04-25T04:46:00.862 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111037.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111037.log.gz 2024-04-25T04:46:00.862 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107411.log 2024-04-25T04:46:00.863 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116680.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116680.log.gz 2024-04-25T04:46:00.863 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50563.log 2024-04-25T04:46:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107411.log.gz 2024-04-25T04:46:00.864 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116727.log 2024-04-25T04:46:00.865 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50563.log.gz 2024-04-25T04:46:00.865 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129498.log 2024-04-25T04:46:00.866 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116727.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116727.log.gz 2024-04-25T04:46:00.866 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119378.log 2024-04-25T04:46:00.867 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129498.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129498.log.gz 2024-04-25T04:46:00.867 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117860.log 2024-04-25T04:46:00.868 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119378.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119378.log.gz 2024-04-25T04:46:00.868 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44420.log 2024-04-25T04:46:00.869 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117860.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117860.log.gz 2024-04-25T04:46:00.869 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152550.log 2024-04-25T04:46:00.869 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44420.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44420.log.gz 2024-04-25T04:46:00.870 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145940.log 2024-04-25T04:46:00.870 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152550.log.gz 2024-04-25T04:46:00.871 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65233.log 2024-04-25T04:46:00.871 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145940.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145940.log.gz 2024-04-25T04:46:00.871 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109046.log 2024-04-25T04:46:00.872 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65233.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65233.log.gz 2024-04-25T04:46:00.872 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51838.log 2024-04-25T04:46:00.873 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109046.log.gz 2024-04-25T04:46:00.873 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99387.log 2024-04-25T04:46:00.874 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51838.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51838.log.gz 2024-04-25T04:46:00.874 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67522.log 2024-04-25T04:46:00.875 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99387.log.gz 2024-04-25T04:46:00.875 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107412.log 2024-04-25T04:46:00.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67522.log.gz 2024-04-25T04:46:00.876 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100021.log 2024-04-25T04:46:00.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107412.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107412.log.gz 2024-04-25T04:46:00.877 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93495.log 2024-04-25T04:46:00.878 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100021.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100021.log.gz 2024-04-25T04:46:00.878 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91919.log 2024-04-25T04:46:00.878 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93495.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93495.log.gz 2024-04-25T04:46:00.879 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137722.log 2024-04-25T04:46:00.879 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91919.log.gz 2024-04-25T04:46:00.880 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136611.log 2024-04-25T04:46:00.880 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137722.log.gz 2024-04-25T04:46:00.880 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90146.log 2024-04-25T04:46:00.881 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136611.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136611.log.gz 2024-04-25T04:46:00.881 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77996.log 2024-04-25T04:46:00.882 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90146.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90146.log.gz 2024-04-25T04:46:00.882 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81799.log 2024-04-25T04:46:00.883 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77996.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77996.log.gz 2024-04-25T04:46:00.883 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154807.log 2024-04-25T04:46:00.884 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81799.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81799.log.gz 2024-04-25T04:46:00.884 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41328.log 2024-04-25T04:46:00.885 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154807.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154807.log.gz 2024-04-25T04:46:00.885 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105302.log 2024-04-25T04:46:00.886 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.0.41328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48409.log 2024-04-25T04:46:00.886 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105302.log.gz 2024-04-25T04:46:00.887 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159024.log 2024-04-25T04:46:00.887 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48409.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48409.log.gz 2024-04-25T04:46:00.888 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99785.log 2024-04-25T04:46:00.888 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159024.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159024.log.gz 2024-04-25T04:46:00.889 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123473.log 2024-04-25T04:46:00.889 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99785.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99785.log.gz 2024-04-25T04:46:00.890 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.149202.log 2024-04-25T04:46:00.890 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.123473.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123473.log.gz 2024-04-25T04:46:00.890 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160149.log 2024-04-25T04:46:00.891 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.149202.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.149202.log.gz 2024-04-25T04:46:00.891 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125348.log 2024-04-25T04:46:00.892 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.160149.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160149.log.gz 2024-04-25T04:46:00.892 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100750.log 2024-04-25T04:46:00.893 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125348.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125348.log.gz 2024-04-25T04:46:00.893 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143538.log 2024-04-25T04:46:00.893 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100750.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100750.log.gz 2024-04-25T04:46:00.894 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139473.log 2024-04-25T04:46:00.894 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143538.log.gz 2024-04-25T04:46:00.895 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61636.log 2024-04-25T04:46:00.895 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139473.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139473.log.gz 2024-04-25T04:46:00.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84874.log 2024-04-25T04:46:00.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61636.log.gz 2024-04-25T04:46:00.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157440.log 2024-04-25T04:46:00.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84874.log.gz 2024-04-25T04:46:00.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82933.log 2024-04-25T04:46:00.898 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157440.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157440.log.gz 2024-04-25T04:46:00.898 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152468.log 2024-04-25T04:46:00.899 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82933.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82933.log.gz 2024-04-25T04:46:00.899 INFO:teuthology.orchestra.run.smithi188.stderr: 81.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.41328.log.gz 2024-04-25T04:46:00.899 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140668.log 2024-04-25T04:46:00.900 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152468.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152468.log.gz 2024-04-25T04:46:00.900 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110396.log 2024-04-25T04:46:00.901 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140668.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140668.log.gz 2024-04-25T04:46:00.901 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89159.log 2024-04-25T04:46:00.902 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110396.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110396.log.gz 2024-04-25T04:46:00.902 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81420.log 2024-04-25T04:46:00.903 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89159.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89159.log.gz 2024-04-25T04:46:00.903 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84414.log 2024-04-25T04:46:00.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81420.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81420.log.gz 2024-04-25T04:46:00.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135546.log 2024-04-25T04:46:00.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84414.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84414.log.gz 2024-04-25T04:46:00.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66524.log 2024-04-25T04:46:00.906 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.135546.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135546.log.gz 2024-04-25T04:46:00.906 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102475.log 2024-04-25T04:46:00.906 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66524.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66524.log.gz 2024-04-25T04:46:00.907 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43829.log 2024-04-25T04:46:00.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102475.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102475.log.gz 2024-04-25T04:46:00.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55920.log 2024-04-25T04:46:00.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43829.log.gz 2024-04-25T04:46:00.909 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58101.log 2024-04-25T04:46:00.909 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.55920.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55920.log.gz 2024-04-25T04:46:00.910 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136538.log 2024-04-25T04:46:00.910 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58101.log.gz 2024-04-25T04:46:00.911 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80660.log 2024-04-25T04:46:00.911 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136538.log.gz 2024-04-25T04:46:00.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56290.log 2024-04-25T04:46:00.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80660.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80660.log.gz 2024-04-25T04:46:00.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119336.log 2024-04-25T04:46:00.913 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56290.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56290.log.gz 2024-04-25T04:46:00.913 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80403.log 2024-04-25T04:46:00.914 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119336.log.gz 2024-04-25T04:46:00.914 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121447.log 2024-04-25T04:46:00.915 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80403.log.gz 2024-04-25T04:46:00.915 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84371.log 2024-04-25T04:46:00.916 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121447.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121447.log.gz 2024-04-25T04:46:00.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127331.log 2024-04-25T04:46:00.917 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84371.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84371.log.gz 2024-04-25T04:46:00.917 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81690.log 2024-04-25T04:46:00.918 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127331.log.gz 2024-04-25T04:46:00.918 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51606.log 2024-04-25T04:46:00.918 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81690.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81690.log.gz 2024-04-25T04:46:00.919 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81108.log 2024-04-25T04:46:00.919 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51606.log.gz 2024-04-25T04:46:00.919 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52104.log 2024-04-25T04:46:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81108.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81108.log.gz 2024-04-25T04:46:00.920 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55792.log 2024-04-25T04:46:00.921 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52104.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52104.log.gz 2024-04-25T04:46:00.921 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115816.log 2024-04-25T04:46:00.922 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55792.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55792.log.gz 2024-04-25T04:46:00.922 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63292.log 2024-04-25T04:46:00.923 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115816.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115816.log.gz 2024-04-25T04:46:00.923 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72156.log 2024-04-25T04:46:00.924 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63292.log.gz 2024-04-25T04:46:00.924 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141506.log 2024-04-25T04:46:00.924 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72156.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72156.log.gz 2024-04-25T04:46:00.925 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117654.log 2024-04-25T04:46:00.926 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141506.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141506.log.gz 2024-04-25T04:46:00.926 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96760.log 2024-04-25T04:46:00.926 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117654.log.gz 2024-04-25T04:46:00.927 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148418.log 2024-04-25T04:46:00.927 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96760.log.gz 2024-04-25T04:46:00.927 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153477.log 2024-04-25T04:46:00.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148418.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148418.log.gz 2024-04-25T04:46:00.928 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152870.log 2024-04-25T04:46:00.929 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153477.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153477.log.gz 2024-04-25T04:46:00.929 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154436.log 2024-04-25T04:46:00.930 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152870.log.gz 2024-04-25T04:46:00.930 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117962.log 2024-04-25T04:46:00.930 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154436.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154436.log.gz 2024-04-25T04:46:00.931 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147523.log 2024-04-25T04:46:00.931 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117962.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117962.log.gz 2024-04-25T04:46:00.932 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150240.log 2024-04-25T04:46:00.933 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147523.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147523.log.gz 2024-04-25T04:46:00.933 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64391.log 2024-04-25T04:46:00.933 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150240.log.gz 2024-04-25T04:46:00.934 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153205.log 2024-04-25T04:46:00.934 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64391.log.gz 2024-04-25T04:46:00.935 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60316.log 2024-04-25T04:46:00.935 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153205.log.gz 2024-04-25T04:46:00.936 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151306.log 2024-04-25T04:46:00.936 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60316.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60316.log.gz 2024-04-25T04:46:00.936 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91495.log 2024-04-25T04:46:00.937 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151306.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151306.log.gz 2024-04-25T04:46:00.937 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113673.log 2024-04-25T04:46:00.938 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91495.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91495.log.gz 2024-04-25T04:46:00.938 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158440.log 2024-04-25T04:46:00.939 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113673.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113673.log.gz 2024-04-25T04:46:00.939 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45435.log 2024-04-25T04:46:00.940 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158440.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158440.log.gz 2024-04-25T04:46:00.940 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53830.log 2024-04-25T04:46:00.940 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45435.log.gz 2024-04-25T04:46:00.941 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92229.log 2024-04-25T04:46:00.941 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53830.log.gz 2024-04-25T04:46:00.941 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142642.log 2024-04-25T04:46:00.942 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92229.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92229.log.gz 2024-04-25T04:46:00.942 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60254.log 2024-04-25T04:46:00.943 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142642.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142642.log.gz 2024-04-25T04:46:00.943 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83080.log 2024-04-25T04:46:00.944 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60254.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60254.log.gz 2024-04-25T04:46:00.944 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156048.log 2024-04-25T04:46:00.945 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83080.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83080.log.gz 2024-04-25T04:46:00.945 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97180.log 2024-04-25T04:46:00.945 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156048.log.gz 2024-04-25T04:46:00.946 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147799.log 2024-04-25T04:46:00.947 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97180.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97180.log.gz 2024-04-25T04:46:00.947 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71520.log 2024-04-25T04:46:00.947 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.147799.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147799.log.gz 2024-04-25T04:46:00.948 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104841.log 2024-04-25T04:46:00.948 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71520.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71520.log.gz 2024-04-25T04:46:00.949 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66610.log 2024-04-25T04:46:00.949 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104841.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104841.log.gz 2024-04-25T04:46:00.949 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72749.log 2024-04-25T04:46:00.950 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66610.log.gz 2024-04-25T04:46:00.950 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57873.log 2024-04-25T04:46:00.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72749.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72749.log.gz 2024-04-25T04:46:00.951 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148182.log 2024-04-25T04:46:00.952 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57873.log.gz 2024-04-25T04:46:00.952 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78147.log 2024-04-25T04:46:00.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148182.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148182.log.gz 2024-04-25T04:46:00.953 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117927.log 2024-04-25T04:46:00.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78147.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78147.log.gz 2024-04-25T04:46:00.953 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118468.log 2024-04-25T04:46:00.955 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117927.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117927.log.gz 2024-04-25T04:46:00.955 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156302.log 2024-04-25T04:46:00.955 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118468.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118468.log.gz 2024-04-25T04:46:00.956 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131660.log 2024-04-25T04:46:00.956 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156302.log.gz 2024-04-25T04:46:00.956 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142203.log 2024-04-25T04:46:00.957 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131660.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131660.log.gz 2024-04-25T04:46:00.957 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113087.log 2024-04-25T04:46:00.958 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142203.log.gz 2024-04-25T04:46:00.958 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103890.log 2024-04-25T04:46:00.959 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113087.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113087.log.gz 2024-04-25T04:46:00.959 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141801.log 2024-04-25T04:46:00.959 INFO:teuthology.orchestra.run.smithi173.stderr: 92.3% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-04-25T04:46:00.960 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103890.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103890.log.gz 2024-04-25T04:46:00.960 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149250.log 2024-04-25T04:46:00.961 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141801.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141801.log.gz 2024-04-25T04:46:00.961 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78342.log 2024-04-25T04:46:00.961 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149250.log: gzip -5 --verbose 0.0%-- /var/log/ceph/cluster1-client.admin.149068.log -- replaced with /var/log/ceph/cluster1-client.mirror.149250.log.gz 2024-04-25T04:46:00.962 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-25T04:46:00.962 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78342.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78342.log.gz 2024-04-25T04:46:00.962 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59690.log 2024-04-25T04:46:00.963 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.149068.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.149068.log.gz 2024-04-25T04:46:00.963 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152133.log 2024-04-25T04:46:00.964 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59690.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59690.log.gz 2024-04-25T04:46:00.964 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56038.log 2024-04-25T04:46:00.965 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152133.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152133.log.gz 2024-04-25T04:46:00.965 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131440.log 2024-04-25T04:46:00.966 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56038.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56038.log.gz 2024-04-25T04:46:00.966 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119463.log 2024-04-25T04:46:00.967 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131440.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131440.log.gz 2024-04-25T04:46:00.967 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97652.log 2024-04-25T04:46:00.968 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119463.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119463.log.gz 2024-04-25T04:46:00.968 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68509.log 2024-04-25T04:46:00.968 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97652.log.gz 2024-04-25T04:46:00.969 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114071.log 2024-04-25T04:46:00.969 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68509.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68509.log.gz 2024-04-25T04:46:00.970 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36647.log 2024-04-25T04:46:00.970 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114071.log.gz 2024-04-25T04:46:00.971 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143412.log 2024-04-25T04:46:00.971 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.36647.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36647.log.gz 2024-04-25T04:46:00.972 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132848.log 2024-04-25T04:46:00.972 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143412.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143412.log.gz 2024-04-25T04:46:00.973 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105638.log 2024-04-25T04:46:00.973 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132848.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132848.log.gz 2024-04-25T04:46:00.974 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87960.log 2024-04-25T04:46:00.974 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105638.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105638.log.gz 2024-04-25T04:46:00.974 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123205.log 2024-04-25T04:46:00.975 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87960.log.gz 2024-04-25T04:46:00.975 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140197.log 2024-04-25T04:46:00.976 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123205.log.gz 2024-04-25T04:46:00.976 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80613.log 2024-04-25T04:46:00.977 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140197.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140197.log.gz 2024-04-25T04:46:00.977 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81900.log 2024-04-25T04:46:00.977 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80613.log.gz 2024-04-25T04:46:00.978 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53365.log 2024-04-25T04:46:00.979 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81900.log.gz 2024-04-25T04:46:00.979 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63606.log 2024-04-25T04:46:00.980 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53365.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53365.log.gz 2024-04-25T04:46:00.980 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100552.log 2024-04-25T04:46:00.980 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63606.log.gz 2024-04-25T04:46:00.980 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72998.log 2024-04-25T04:46:00.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100552.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100552.log.gz 2024-04-25T04:46:00.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70973.log 2024-04-25T04:46:00.982 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72998.log.gz 2024-04-25T04:46:00.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141445.log 2024-04-25T04:46:00.983 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70973.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70973.log.gz 2024-04-25T04:46:00.983 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93350.log 2024-04-25T04:46:00.984 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141445.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141445.log.gz 2024-04-25T04:46:00.984 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130215.log 2024-04-25T04:46:00.985 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93350.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93350.log.gz 2024-04-25T04:46:00.985 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51106.log 2024-04-25T04:46:00.986 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130215.log.gz 2024-04-25T04:46:00.986 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103095.log 2024-04-25T04:46:00.987 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51106.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51106.log.gz 2024-04-25T04:46:00.987 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83081.log 2024-04-25T04:46:00.988 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103095.log.gz 2024-04-25T04:46:00.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106040.log 2024-04-25T04:46:00.989 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83081.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83081.log.gz 2024-04-25T04:46:00.989 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90725.log 2024-04-25T04:46:00.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106040.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106040.log.gz 2024-04-25T04:46:00.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134665.log 2024-04-25T04:46:00.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90725.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90725.log.gz 2024-04-25T04:46:00.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96215.log 2024-04-25T04:46:00.991 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134665.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134665.log.gz 2024-04-25T04:46:00.992 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145610.log 2024-04-25T04:46:00.992 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96215.log.gz 2024-04-25T04:46:00.992 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125328.log 2024-04-25T04:46:00.993 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145610.log.gz 2024-04-25T04:46:00.993 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51855.log 2024-04-25T04:46:00.994 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125328.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125328.log.gz 2024-04-25T04:46:00.994 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57576.log 2024-04-25T04:46:00.995 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51855.log.gz 2024-04-25T04:46:00.995 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84747.log 2024-04-25T04:46:00.995 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57576.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57576.log.gz 2024-04-25T04:46:00.996 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55153.log 2024-04-25T04:46:00.997 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84747.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84747.log.gz 2024-04-25T04:46:00.997 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84204.log 2024-04-25T04:46:00.998 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55153.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55153.log.gz 2024-04-25T04:46:00.998 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145042.log 2024-04-25T04:46:00.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84204.log.gz 2024-04-25T04:46:00.999 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103124.log 2024-04-25T04:46:00.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.145042.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145042.log.gz 2024-04-25T04:46:01.000 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104883.log 2024-04-25T04:46:01.000 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103124.log.gz 2024-04-25T04:46:01.001 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73118.log 2024-04-25T04:46:01.001 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104883.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104883.log.gz 2024-04-25T04:46:01.001 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140456.log 2024-04-25T04:46:01.002 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73118.log.gz 2024-04-25T04:46:01.002 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159941.log 2024-04-25T04:46:01.003 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140456.log.gz 2024-04-25T04:46:01.003 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147981.log 2024-04-25T04:46:01.004 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159941.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159941.log.gz 2024-04-25T04:46:01.004 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44798.log 2024-04-25T04:46:01.005 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147981.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147981.log.gz 2024-04-25T04:46:01.005 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121458.log 2024-04-25T04:46:01.005 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44798.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44798.log.gz 2024-04-25T04:46:01.006 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75747.log 2024-04-25T04:46:01.007 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121458.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121458.log.gz 2024-04-25T04:46:01.007 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48664.log 2024-04-25T04:46:01.007 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75747.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75747.log.gz 2024-04-25T04:46:01.008 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49246.log 2024-04-25T04:46:01.008 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48664.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48664.log.gz 2024-04-25T04:46:01.008 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136813.log 2024-04-25T04:46:01.009 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49246.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49246.log.gz 2024-04-25T04:46:01.009 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60042.log 2024-04-25T04:46:01.010 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136813.log.gz 2024-04-25T04:46:01.011 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149781.log 2024-04-25T04:46:01.011 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60042.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60042.log.gz 2024-04-25T04:46:01.011 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82306.log 2024-04-25T04:46:01.012 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149781.log.gz 2024-04-25T04:46:01.012 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132890.log 2024-04-25T04:46:01.013 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82306.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82306.log.gz 2024-04-25T04:46:01.013 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77406.log 2024-04-25T04:46:01.014 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132890.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132890.log.gz 2024-04-25T04:46:01.014 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46575.log 2024-04-25T04:46:01.014 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77406.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77406.log.gz 2024-04-25T04:46:01.015 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48997.log 2024-04-25T04:46:01.016 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46575.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46575.log.gz 2024-04-25T04:46:01.016 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112080.log 2024-04-25T04:46:01.016 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48997.log.gz 2024-04-25T04:46:01.017 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92523.log 2024-04-25T04:46:01.017 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112080.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112080.log.gz 2024-04-25T04:46:01.017 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53432.log 2024-04-25T04:46:01.019 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92523.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92523.log.gz 2024-04-25T04:46:01.019 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89314.log 2024-04-25T04:46:01.019 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.53432.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53432.log.gz 2024-04-25T04:46:01.020 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128021.log 2024-04-25T04:46:01.020 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89314.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89314.log.gz 2024-04-25T04:46:01.020 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158659.log 2024-04-25T04:46:01.021 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128021.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128021.log.gz 2024-04-25T04:46:01.021 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141655.log 2024-04-25T04:46:01.022 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158659.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158659.log.gz 2024-04-25T04:46:01.022 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70196.log 2024-04-25T04:46:01.023 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141655.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141655.log.gz 2024-04-25T04:46:01.023 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41144.log 2024-04-25T04:46:01.024 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70196.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70196.log.gz 2024-04-25T04:46:01.024 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79341.log 2024-04-25T04:46:01.025 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41144.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41144.log.gz 2024-04-25T04:46:01.025 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145434.log 2024-04-25T04:46:01.026 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79341.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79341.log.gz 2024-04-25T04:46:01.026 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88421.log 2024-04-25T04:46:01.026 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145434.log.gz 2024-04-25T04:46:01.027 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97306.log 2024-04-25T04:46:01.027 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88421.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88421.log.gz 2024-04-25T04:46:01.028 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107518.log 2024-04-25T04:46:01.028 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97306.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97306.log.gz 2024-04-25T04:46:01.028 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44105.log 2024-04-25T04:46:01.029 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107518.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107518.log.gz 2024-04-25T04:46:01.029 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111839.log 2024-04-25T04:46:01.030 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44105.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44105.log.gz 2024-04-25T04:46:01.030 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156634.log 2024-04-25T04:46:01.031 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111839.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111839.log.gz 2024-04-25T04:46:01.031 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46129.log 2024-04-25T04:46:01.032 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156634.log.gz 2024-04-25T04:46:01.032 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72450.log 2024-04-25T04:46:01.033 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46129.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46129.log.gz 2024-04-25T04:46:01.033 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134859.log 2024-04-25T04:46:01.034 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72450.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72450.log.gz 2024-04-25T04:46:01.034 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53970.log 2024-04-25T04:46:01.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134859.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134859.log.gz 2024-04-25T04:46:01.035 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87076.log 2024-04-25T04:46:01.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53970.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53970.log.gz 2024-04-25T04:46:01.036 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150552.log 2024-04-25T04:46:01.036 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87076.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87076.log.gz 2024-04-25T04:46:01.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91033.log 2024-04-25T04:46:01.037 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150552.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150552.log.gz 2024-04-25T04:46:01.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.122755.log 2024-04-25T04:46:01.038 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91033.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91033.log.gz 2024-04-25T04:46:01.038 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84833.log 2024-04-25T04:46:01.039 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.122755.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.122755.log.gz 2024-04-25T04:46:01.039 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150266.log 2024-04-25T04:46:01.040 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84833.log.gz 2024-04-25T04:46:01.040 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113315.log 2024-04-25T04:46:01.041 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150266.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150266.log.gz 2024-04-25T04:46:01.041 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152910.log 2024-04-25T04:46:01.042 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113315.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113315.log.gz 2024-04-25T04:46:01.042 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95500.log 2024-04-25T04:46:01.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152910.log.gz 2024-04-25T04:46:01.043 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119862.log 2024-04-25T04:46:01.044 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95500.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95500.log.gz 2024-04-25T04:46:01.044 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48030.log 2024-04-25T04:46:01.045 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119862.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119862.log.gz 2024-04-25T04:46:01.045 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79077.log 2024-04-25T04:46:01.046 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48030.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48030.log.gz 2024-04-25T04:46:01.046 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147779.log 2024-04-25T04:46:01.047 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79077.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79077.log.gz 2024-04-25T04:46:01.047 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46215.log 2024-04-25T04:46:01.048 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.147779.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147779.log.gz 2024-04-25T04:46:01.048 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95857.log 2024-04-25T04:46:01.049 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46215.log.gz 2024-04-25T04:46:01.049 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51247.log 2024-04-25T04:46:01.050 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95857.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95857.log.gz 2024-04-25T04:46:01.050 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118743.log 2024-04-25T04:46:01.051 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51247.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51247.log.gz 2024-04-25T04:46:01.051 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105762.log 2024-04-25T04:46:01.052 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118743.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118743.log.gz 2024-04-25T04:46:01.052 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125539.log 2024-04-25T04:46:01.053 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105762.log.gz 2024-04-25T04:46:01.053 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128680.log 2024-04-25T04:46:01.054 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125539.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125539.log.gz 2024-04-25T04:46:01.054 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86491.log 2024-04-25T04:46:01.055 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128680.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128680.log.gz 2024-04-25T04:46:01.055 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59680.log 2024-04-25T04:46:01.055 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86491.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86491.log.gz 2024-04-25T04:46:01.056 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110076.log 2024-04-25T04:46:01.057 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59680.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59680.log.gz 2024-04-25T04:46:01.057 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133736.log 2024-04-25T04:46:01.058 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110076.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110076.log.gz 2024-04-25T04:46:01.058 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58291.log 2024-04-25T04:46:01.058 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133736.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133736.log.gz 2024-04-25T04:46:01.059 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127007.log 2024-04-25T04:46:01.059 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58291.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58291.log.gz 2024-04-25T04:46:01.060 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68827.log 2024-04-25T04:46:01.061 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127007.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127007.log.gz 2024-04-25T04:46:01.061 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95093.log 2024-04-25T04:46:01.061 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68827.log.gz 2024-04-25T04:46:01.062 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78858.log 2024-04-25T04:46:01.062 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95093.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95093.log.gz 2024-04-25T04:46:01.063 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43954.log 2024-04-25T04:46:01.064 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78858.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78858.log.gz 2024-04-25T04:46:01.064 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47632.log 2024-04-25T04:46:01.064 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43954.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43954.log.gz 2024-04-25T04:46:01.065 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90723.log 2024-04-25T04:46:01.065 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47632.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47632.log.gz 2024-04-25T04:46:01.065 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75131.log 2024-04-25T04:46:01.066 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90723.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90723.log.gz 2024-04-25T04:46:01.066 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65697.log 2024-04-25T04:46:01.067 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75131.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75131.log.gz 2024-04-25T04:46:01.067 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113596.log 2024-04-25T04:46:01.068 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65697.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65697.log.gz 2024-04-25T04:46:01.068 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131606.log 2024-04-25T04:46:01.069 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113596.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113596.log.gz 2024-04-25T04:46:01.069 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89037.log 2024-04-25T04:46:01.070 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131606.log.gz 2024-04-25T04:46:01.070 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58459.log 2024-04-25T04:46:01.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89037.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89037.log.gz 2024-04-25T04:46:01.071 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108796.log 2024-04-25T04:46:01.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58459.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58459.log.gz 2024-04-25T04:46:01.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115543.log 2024-04-25T04:46:01.073 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108796.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108796.log.gz 2024-04-25T04:46:01.073 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89501.log 2024-04-25T04:46:01.074 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115543.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115543.log.gz 2024-04-25T04:46:01.074 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68257.log 2024-04-25T04:46:01.075 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89501.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89501.log.gz 2024-04-25T04:46:01.075 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115856.log 2024-04-25T04:46:01.076 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68257.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68257.log.gz 2024-04-25T04:46:01.076 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46487.log 2024-04-25T04:46:01.077 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115856.log.gz 2024-04-25T04:46:01.077 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44901.log 2024-04-25T04:46:01.078 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46487.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46487.log.gz 2024-04-25T04:46:01.078 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93285.log 2024-04-25T04:46:01.079 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44901.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44901.log.gz 2024-04-25T04:46:01.079 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149779.log 2024-04-25T04:46:01.080 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93285.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93285.log.gz 2024-04-25T04:46:01.080 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116068.log 2024-04-25T04:46:01.081 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149779.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149779.log.gz 2024-04-25T04:46:01.081 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113114.log 2024-04-25T04:46:01.082 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116068.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116068.log.gz 2024-04-25T04:46:01.082 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106732.log 2024-04-25T04:46:01.083 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113114.log.gz 2024-04-25T04:46:01.084 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97527.log 2024-04-25T04:46:01.084 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106732.log.gz 2024-04-25T04:46:01.084 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110806.log 2024-04-25T04:46:01.085 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97527.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97527.log.gz 2024-04-25T04:46:01.086 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92544.log 2024-04-25T04:46:01.086 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110806.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110806.log.gz 2024-04-25T04:46:01.086 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92083.log 2024-04-25T04:46:01.087 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92544.log.gz 2024-04-25T04:46:01.088 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73451.log 2024-04-25T04:46:01.088 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92083.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92083.log.gz 2024-04-25T04:46:01.088 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137041.log 2024-04-25T04:46:01.089 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73451.log.gz 2024-04-25T04:46:01.090 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63981.log 2024-04-25T04:46:01.090 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137041.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137041.log.gz 2024-04-25T04:46:01.090 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160318.log 2024-04-25T04:46:01.091 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63981.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63981.log.gz 2024-04-25T04:46:01.091 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123131.log 2024-04-25T04:46:01.092 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160318.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160318.log.gz 2024-04-25T04:46:01.092 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100020.log 2024-04-25T04:46:01.093 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123131.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123131.log.gz 2024-04-25T04:46:01.093 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75211.log 2024-04-25T04:46:01.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100020.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100020.log.gz 2024-04-25T04:46:01.094 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123520.log 2024-04-25T04:46:01.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75211.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75211.log.gz 2024-04-25T04:46:01.095 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123001.log 2024-04-25T04:46:01.096 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123520.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123520.log.gz 2024-04-25T04:46:01.096 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144082.log 2024-04-25T04:46:01.097 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123001.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123001.log.gz 2024-04-25T04:46:01.097 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68593.log 2024-04-25T04:46:01.098 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144082.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144082.log.gz 2024-04-25T04:46:01.098 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155271.log 2024-04-25T04:46:01.099 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68593.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68593.log.gz 2024-04-25T04:46:01.099 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123686.log 2024-04-25T04:46:01.100 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155271.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155271.log.gz 2024-04-25T04:46:01.100 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142287.log 2024-04-25T04:46:01.101 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123686.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123686.log.gz 2024-04-25T04:46:01.101 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83529.log 2024-04-25T04:46:01.102 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142287.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142287.log.gz 2024-04-25T04:46:01.102 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106942.log 2024-04-25T04:46:01.103 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83529.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83529.log.gz 2024-04-25T04:46:01.103 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81169.log 2024-04-25T04:46:01.104 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106942.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106942.log.gz 2024-04-25T04:46:01.104 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96845.log 2024-04-25T04:46:01.105 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81169.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81169.log.gz 2024-04-25T04:46:01.105 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.98911.log 2024-04-25T04:46:01.106 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96845.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96845.log.gz 2024-04-25T04:46:01.106 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51025.log 2024-04-25T04:46:01.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.98911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.98911.log.gz 2024-04-25T04:46:01.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115650.log 2024-04-25T04:46:01.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51025.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51025.log.gz 2024-04-25T04:46:01.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92822.log 2024-04-25T04:46:01.109 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115650.log.gz 2024-04-25T04:46:01.109 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48534.log 2024-04-25T04:46:01.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92822.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92822.log.gz 2024-04-25T04:46:01.110 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137676.log 2024-04-25T04:46:01.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48534.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48534.log.gz 2024-04-25T04:46:01.111 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127602.log 2024-04-25T04:46:01.111 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137676.log.gz 2024-04-25T04:46:01.112 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130170.log 2024-04-25T04:46:01.112 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127602.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127602.log.gz 2024-04-25T04:46:01.113 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71699.log 2024-04-25T04:46:01.113 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130170.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130170.log.gz 2024-04-25T04:46:01.114 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159672.log 2024-04-25T04:46:01.114 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71699.log.gz 2024-04-25T04:46:01.115 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137506.log 2024-04-25T04:46:01.115 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159672.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159672.log.gz 2024-04-25T04:46:01.116 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108356.log 2024-04-25T04:46:01.116 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137506.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137506.log.gz 2024-04-25T04:46:01.117 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94673.log 2024-04-25T04:46:01.118 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108356.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108356.log.gz 2024-04-25T04:46:01.118 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59937.log 2024-04-25T04:46:01.118 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94673.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94673.log.gz 2024-04-25T04:46:01.119 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103469.log 2024-04-25T04:46:01.119 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59937.log.gz 2024-04-25T04:46:01.120 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79011.log 2024-04-25T04:46:01.120 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103469.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103469.log.gz 2024-04-25T04:46:01.121 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135247.log 2024-04-25T04:46:01.121 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79011.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79011.log.gz 2024-04-25T04:46:01.121 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132594.log 2024-04-25T04:46:01.122 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135247.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135247.log.gz 2024-04-25T04:46:01.123 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54949.log 2024-04-25T04:46:01.123 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132594.log.gz 2024-04-25T04:46:01.123 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152049.log 2024-04-25T04:46:01.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54949.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54949.log.gz 2024-04-25T04:46:01.125 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75214.log 2024-04-25T04:46:01.125 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152049.log.gz 2024-04-25T04:46:01.126 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61240.log 2024-04-25T04:46:01.126 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75214.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75214.log.gz 2024-04-25T04:46:01.127 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129754.log 2024-04-25T04:46:01.127 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61240.log.gz 2024-04-25T04:46:01.128 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46806.log 2024-04-25T04:46:01.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129754.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129754.log.gz 2024-04-25T04:46:01.129 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131607.log 2024-04-25T04:46:01.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46806.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46806.log.gz 2024-04-25T04:46:01.130 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118255.log 2024-04-25T04:46:01.130 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131607.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131607.log.gz 2024-04-25T04:46:01.130 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128952.log 2024-04-25T04:46:01.132 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138519.log 2024-04-25T04:46:01.132 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118255.log.gz 2024-04-25T04:46:01.132 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128952.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128952.log.gz 2024-04-25T04:46:01.132 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44504.log 2024-04-25T04:46:01.133 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138519.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138519.log.gz 2024-04-25T04:46:01.133 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75590.log 2024-04-25T04:46:01.134 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44504.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44504.log.gz 2024-04-25T04:46:01.134 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85109.log 2024-04-25T04:46:01.135 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75590.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75590.log.gz 2024-04-25T04:46:01.136 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69816.log 2024-04-25T04:46:01.136 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85109.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85109.log.gz 2024-04-25T04:46:01.136 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61699.log 2024-04-25T04:46:01.137 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69816.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69816.log.gz 2024-04-25T04:46:01.137 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120006.log 2024-04-25T04:46:01.138 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61699.log.gz 2024-04-25T04:46:01.138 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65530.log 2024-04-25T04:46:01.139 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120006.log.gz 2024-04-25T04:46:01.139 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99172.log 2024-04-25T04:46:01.140 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65530.log.gz 2024-04-25T04:46:01.140 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77836.log 2024-04-25T04:46:01.141 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99172.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99172.log.gz 2024-04-25T04:46:01.141 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88725.log 2024-04-25T04:46:01.142 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77836.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77836.log.gz 2024-04-25T04:46:01.143 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93328.log 2024-04-25T04:46:01.143 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88725.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88725.log.gz 2024-04-25T04:46:01.144 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101120.log 2024-04-25T04:46:01.144 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93328.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93328.log.gz 2024-04-25T04:46:01.144 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105514.log 2024-04-25T04:46:01.145 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101120.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101120.log.gz 2024-04-25T04:46:01.145 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44379.log 2024-04-25T04:46:01.146 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105514.log.gz 2024-04-25T04:46:01.146 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90416.log 2024-04-25T04:46:01.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44379.log.gz 2024-04-25T04:46:01.147 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51899.log 2024-04-25T04:46:01.148 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90416.log.gz 2024-04-25T04:46:01.148 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62085.log 2024-04-25T04:46:01.149 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51899.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51899.log.gz 2024-04-25T04:46:01.149 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84266.log 2024-04-25T04:46:01.150 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62085.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62085.log.gz 2024-04-25T04:46:01.150 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143453.log 2024-04-25T04:46:01.151 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84266.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84266.log.gz 2024-04-25T04:46:01.151 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148512.log 2024-04-25T04:46:01.152 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143453.log.gz 2024-04-25T04:46:01.152 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66267.log 2024-04-25T04:46:01.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148512.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148512.log.gz 2024-04-25T04:46:01.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70762.log 2024-04-25T04:46:01.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.66267.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66267.log.gz 2024-04-25T04:46:01.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103786.log 2024-04-25T04:46:01.155 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70762.log.gz 2024-04-25T04:46:01.155 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107724.log 2024-04-25T04:46:01.156 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103786.log.gz 2024-04-25T04:46:01.156 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77333.log 2024-04-25T04:46:01.156 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107724.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107724.log.gz 2024-04-25T04:46:01.157 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129119.log 2024-04-25T04:46:01.157 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77333.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77333.log.gz 2024-04-25T04:46:01.158 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46665.log 2024-04-25T04:46:01.159 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129119.log.gz 2024-04-25T04:46:01.159 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125519.log 2024-04-25T04:46:01.159 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46665.log.gz 2024-04-25T04:46:01.160 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129856.log 2024-04-25T04:46:01.161 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125519.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125519.log.gz 2024-04-25T04:46:01.161 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95787.log 2024-04-25T04:46:01.162 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129856.log.gz 2024-04-25T04:46:01.162 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77056.log 2024-04-25T04:46:01.162 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95787.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95787.log.gz 2024-04-25T04:46:01.162 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141090.log 2024-04-25T04:46:01.163 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77056.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77056.log.gz 2024-04-25T04:46:01.163 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148209.log 2024-04-25T04:46:01.164 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141090.log.gz 2024-04-25T04:46:01.164 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61790.log 2024-04-25T04:46:01.165 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148209.log.gz 2024-04-25T04:46:01.165 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107830.log 2024-04-25T04:46:01.166 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61790.log.gz 2024-04-25T04:46:01.166 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114829.log 2024-04-25T04:46:01.167 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107830.log.gz 2024-04-25T04:46:01.168 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85002.log 2024-04-25T04:46:01.168 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114829.log.gz 2024-04-25T04:46:01.169 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77144.log 2024-04-25T04:46:01.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85002.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85002.log.gz 2024-04-25T04:46:01.170 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113828.log 2024-04-25T04:46:01.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77144.log.gz 2024-04-25T04:46:01.170 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77950.log 2024-04-25T04:46:01.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113828.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113828.log.gz 2024-04-25T04:46:01.171 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160291.log 2024-04-25T04:46:01.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77950.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77950.log.gz 2024-04-25T04:46:01.172 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66059.log 2024-04-25T04:46:01.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160291.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160291.log.gz 2024-04-25T04:46:01.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91767.log 2024-04-25T04:46:01.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.66059.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66059.log.gz 2024-04-25T04:46:01.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103512.log 2024-04-25T04:46:01.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91767.log.gz 2024-04-25T04:46:01.175 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123476.log 2024-04-25T04:46:01.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103512.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103512.log.gz 2024-04-25T04:46:01.176 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79465.log 2024-04-25T04:46:01.177 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.123476.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123476.log.gz 2024-04-25T04:46:01.177 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146614.log 2024-04-25T04:46:01.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.79465.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79465.log.gz 2024-04-25T04:46:01.178 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68405.log 2024-04-25T04:46:01.179 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.146614.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146614.log.gz 2024-04-25T04:46:01.179 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119379.log 2024-04-25T04:46:01.180 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68405.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68405.log.gz 2024-04-25T04:46:01.180 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51216.log 2024-04-25T04:46:01.181 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119379.log.gz 2024-04-25T04:46:01.181 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111841.log 2024-04-25T04:46:01.182 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51216.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51216.log.gz 2024-04-25T04:46:01.182 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60230.log 2024-04-25T04:46:01.183 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111841.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111841.log.gz 2024-04-25T04:46:01.183 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92375.log 2024-04-25T04:46:01.184 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60230.log.gz 2024-04-25T04:46:01.184 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114891.log 2024-04-25T04:46:01.185 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92375.log.gz 2024-04-25T04:46:01.185 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157836.log 2024-04-25T04:46:01.186 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114891.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114891.log.gz 2024-04-25T04:46:01.186 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126015.log 2024-04-25T04:46:01.187 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157836.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157836.log.gz 2024-04-25T04:46:01.187 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71563.log 2024-04-25T04:46:01.188 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126015.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126015.log.gz 2024-04-25T04:46:01.188 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100196.log 2024-04-25T04:46:01.188 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71563.log.gz 2024-04-25T04:46:01.189 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64872.log 2024-04-25T04:46:01.190 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100196.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100196.log.gz 2024-04-25T04:46:01.190 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150613.log 2024-04-25T04:46:01.191 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64872.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64872.log.gz 2024-04-25T04:46:01.191 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69506.log 2024-04-25T04:46:01.192 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150613.log.gz 2024-04-25T04:46:01.192 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104017.log 2024-04-25T04:46:01.193 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69506.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69506.log.gz 2024-04-25T04:46:01.193 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151158.log 2024-04-25T04:46:01.193 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104017.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104017.log.gz 2024-04-25T04:46:01.194 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62274.log 2024-04-25T04:46:01.194 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151158.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151158.log.gz 2024-04-25T04:46:01.195 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51083.log 2024-04-25T04:46:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62274.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62274.log.gz 2024-04-25T04:46:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55856.log 2024-04-25T04:46:01.196 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51083.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51083.log.gz 2024-04-25T04:46:01.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77117.log 2024-04-25T04:46:01.197 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55856.log.gz 2024-04-25T04:46:01.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120365.log 2024-04-25T04:46:01.198 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77117.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77117.log.gz 2024-04-25T04:46:01.198 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151601.log 2024-04-25T04:46:01.199 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120365.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120365.log.gz 2024-04-25T04:46:01.199 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113851.log 2024-04-25T04:46:01.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151601.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151601.log.gz 2024-04-25T04:46:01.200 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139579.log 2024-04-25T04:46:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113851.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113851.log.gz 2024-04-25T04:46:01.201 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135577.log 2024-04-25T04:46:01.202 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139579.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139579.log.gz 2024-04-25T04:46:01.202 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141173.log 2024-04-25T04:46:01.203 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135577.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135577.log.gz 2024-04-25T04:46:01.203 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149456.log 2024-04-25T04:46:01.204 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141173.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141173.log.gz 2024-04-25T04:46:01.204 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99607.log 2024-04-25T04:46:01.205 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149456.log.gz 2024-04-25T04:46:01.205 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120786.log 2024-04-25T04:46:01.206 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99607.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99607.log.gz 2024-04-25T04:46:01.206 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155273.log 2024-04-25T04:46:01.206 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120786.log.gz 2024-04-25T04:46:01.207 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150204.log 2024-04-25T04:46:01.208 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155273.log.gz 2024-04-25T04:46:01.208 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55103.log 2024-04-25T04:46:01.208 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150204.log.gz 2024-04-25T04:46:01.209 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58289.log 2024-04-25T04:46:01.209 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.55103.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55103.log.gz 2024-04-25T04:46:01.209 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60882.log 2024-04-25T04:46:01.210 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58289.log.gz 2024-04-25T04:46:01.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96696.log 2024-04-25T04:46:01.211 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60882.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60882.log.gz 2024-04-25T04:46:01.211 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107242.log 2024-04-25T04:46:01.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96696.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96696.log.gz 2024-04-25T04:46:01.213 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135421.log 2024-04-25T04:46:01.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107242.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107242.log.gz 2024-04-25T04:46:01.214 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131542.log 2024-04-25T04:46:01.214 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.135421.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135421.log.gz 2024-04-25T04:46:01.215 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97658.log 2024-04-25T04:46:01.215 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131542.log.gz 2024-04-25T04:46:01.215 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116977.log 2024-04-25T04:46:01.216 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97658.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97658.log.gz 2024-04-25T04:46:01.216 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91707.log 2024-04-25T04:46:01.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116977.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116977.log.gz 2024-04-25T04:46:01.218 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50027.log 2024-04-25T04:46:01.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91707.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91707.log.gz 2024-04-25T04:46:01.219 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54125.log 2024-04-25T04:46:01.219 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.50027.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50027.log.gz 2024-04-25T04:46:01.219 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137968.log 2024-04-25T04:46:01.220 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54125.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54125.log.gz 2024-04-25T04:46:01.220 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100838.log 2024-04-25T04:46:01.221 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137968.log.gz 2024-04-25T04:46:01.221 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63052.log 2024-04-25T04:46:01.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100838.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100838.log.gz 2024-04-25T04:46:01.223 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87975.log 2024-04-25T04:46:01.223 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63052.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63052.log.gz 2024-04-25T04:46:01.223 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122811.log 2024-04-25T04:46:01.224 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87975.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87975.log.gz 2024-04-25T04:46:01.224 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111332.log 2024-04-25T04:46:01.225 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.122811.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.122811.log.gz 2024-04-25T04:46:01.225 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117438.log 2024-04-25T04:46:01.226 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.111332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111332.log.gz 2024-04-25T04:46:01.226 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67529.log 2024-04-25T04:46:01.227 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117438.log.gz 2024-04-25T04:46:01.227 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122926.log 2024-04-25T04:46:01.228 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67529.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67529.log.gz 2024-04-25T04:46:01.228 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52386.log 2024-04-25T04:46:01.229 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122926.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122926.log.gz 2024-04-25T04:46:01.229 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154293.log 2024-04-25T04:46:01.230 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52386.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52386.log.gz 2024-04-25T04:46:01.230 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122268.log 2024-04-25T04:46:01.231 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154293.log.gz 2024-04-25T04:46:01.231 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118701.log 2024-04-25T04:46:01.232 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122268.log.gz 2024-04-25T04:46:01.232 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135973.log 2024-04-25T04:46:01.233 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118701.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118701.log.gz 2024-04-25T04:46:01.233 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131501.log 2024-04-25T04:46:01.234 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135973.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135973.log.gz 2024-04-25T04:46:01.234 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95732.log 2024-04-25T04:46:01.235 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131501.log.gz 2024-04-25T04:46:01.235 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45226.log 2024-04-25T04:46:01.236 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95732.log.gz 2024-04-25T04:46:01.236 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51546.log 2024-04-25T04:46:01.237 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45226.log.gz 2024-04-25T04:46:01.237 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91071.log 2024-04-25T04:46:01.238 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51546.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51546.log.gz 2024-04-25T04:46:01.238 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156091.log 2024-04-25T04:46:01.239 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91071.log.gz 2024-04-25T04:46:01.239 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118992.log 2024-04-25T04:46:01.240 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156091.log.gz 2024-04-25T04:46:01.240 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46339.log 2024-04-25T04:46:01.241 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118992.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118992.log.gz 2024-04-25T04:46:01.241 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118909.log 2024-04-25T04:46:01.242 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46339.log.gz 2024-04-25T04:46:01.242 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156256.log 2024-04-25T04:46:01.243 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118909.log.gz 2024-04-25T04:46:01.243 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131335.log 2024-04-25T04:46:01.244 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156256.log.gz 2024-04-25T04:46:01.244 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74713.log 2024-04-25T04:46:01.245 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131335.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131335.log.gz 2024-04-25T04:46:01.245 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44545.log 2024-04-25T04:46:01.246 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74713.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74713.log.gz 2024-04-25T04:46:01.246 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103594.log 2024-04-25T04:46:01.247 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44545.log.gz 2024-04-25T04:46:01.247 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127244.log 2024-04-25T04:46:01.248 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103594.log.gz 2024-04-25T04:46:01.248 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99999.log 2024-04-25T04:46:01.249 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127244.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127244.log.gz 2024-04-25T04:46:01.249 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150377.log 2024-04-25T04:46:01.250 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99999.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99999.log.gz 2024-04-25T04:46:01.250 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118051.log 2024-04-25T04:46:01.251 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150377.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150377.log.gz 2024-04-25T04:46:01.251 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129668.log 2024-04-25T04:46:01.252 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159713.log 2024-04-25T04:46:01.252 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118051.log.gz 2024-04-25T04:46:01.253 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129668.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129668.log.gz 2024-04-25T04:46:01.253 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137067.log 2024-04-25T04:46:01.253 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159713.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159713.log.gz 2024-04-25T04:46:01.254 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82348.log 2024-04-25T04:46:01.254 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.137067.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.137067.log.gz 2024-04-25T04:46:01.255 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101973.log 2024-04-25T04:46:01.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82348.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82348.log.gz 2024-04-25T04:46:01.256 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68907.log 2024-04-25T04:46:01.257 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101973.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101973.log.gz 2024-04-25T04:46:01.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80446.log 2024-04-25T04:46:01.257 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68907.log.gz 2024-04-25T04:46:01.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40755.log 2024-04-25T04:46:01.258 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80446.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80446.log.gz 2024-04-25T04:46:01.259 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144058.log 2024-04-25T04:46:01.260 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40755.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40755.log.gz 2024-04-25T04:46:01.260 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98806.log 2024-04-25T04:46:01.260 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144058.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144058.log.gz 2024-04-25T04:46:01.260 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54863.log 2024-04-25T04:46:01.261 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98806.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98806.log.gz 2024-04-25T04:46:01.261 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144187.log 2024-04-25T04:46:01.262 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54863.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54863.log.gz 2024-04-25T04:46:01.263 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133684.log 2024-04-25T04:46:01.263 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144187.log.gz 2024-04-25T04:46:01.264 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101711.log 2024-04-25T04:46:01.264 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133684.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133684.log.gz 2024-04-25T04:46:01.265 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49123.log 2024-04-25T04:46:01.266 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101711.log.gz 2024-04-25T04:46:01.266 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153812.log 2024-04-25T04:46:01.266 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49123.log.gz 2024-04-25T04:46:01.267 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96490.log 2024-04-25T04:46:01.267 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153812.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153812.log.gz 2024-04-25T04:46:01.267 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115228.log 2024-04-25T04:46:01.268 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96490.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96490.log.gz 2024-04-25T04:46:01.268 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111923.log 2024-04-25T04:46:01.269 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115228.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115228.log.gz 2024-04-25T04:46:01.269 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147893.log 2024-04-25T04:46:01.270 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111923.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111923.log.gz 2024-04-25T04:46:01.270 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88553.log 2024-04-25T04:46:01.271 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.147893.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147893.log.gz 2024-04-25T04:46:01.271 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57234.log 2024-04-25T04:46:01.272 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88553.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88553.log.gz 2024-04-25T04:46:01.272 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84098.log 2024-04-25T04:46:01.273 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57234.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57234.log.gz 2024-04-25T04:46:01.274 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96148.log 2024-04-25T04:46:01.274 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84098.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84098.log.gz 2024-04-25T04:46:01.274 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47841.log 2024-04-25T04:46:01.275 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96148.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96148.log.gz 2024-04-25T04:46:01.275 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132191.log 2024-04-25T04:46:01.276 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47841.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47841.log.gz 2024-04-25T04:46:01.276 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49124.log 2024-04-25T04:46:01.277 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132191.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132191.log.gz 2024-04-25T04:46:01.278 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55499.log 2024-04-25T04:46:01.278 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49124.log.gz 2024-04-25T04:46:01.278 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47097.log 2024-04-25T04:46:01.279 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55499.log.gz 2024-04-25T04:46:01.280 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93583.log 2024-04-25T04:46:01.280 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47097.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47097.log.gz 2024-04-25T04:46:01.280 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114985.log 2024-04-25T04:46:01.281 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93583.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93583.log.gz 2024-04-25T04:46:01.281 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106945.log 2024-04-25T04:46:01.282 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114985.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114985.log.gz 2024-04-25T04:46:01.282 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158873.log 2024-04-25T04:46:01.283 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106945.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106945.log.gz 2024-04-25T04:46:01.283 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101767.log 2024-04-25T04:46:01.284 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158873.log.gz 2024-04-25T04:46:01.284 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106483.log 2024-04-25T04:46:01.286 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101767.log.gz 2024-04-25T04:46:01.286 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149448.log 2024-04-25T04:46:01.286 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106483.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106483.log.gz 2024-04-25T04:46:01.287 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56667.log 2024-04-25T04:46:01.287 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149448.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149448.log.gz 2024-04-25T04:46:01.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79586.log 2024-04-25T04:46:01.288 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56667.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56667.log.gz 2024-04-25T04:46:01.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153331.log 2024-04-25T04:46:01.289 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79586.log.gz 2024-04-25T04:46:01.289 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100610.log 2024-04-25T04:46:01.290 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153331.log.gz 2024-04-25T04:46:01.290 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49862.log 2024-04-25T04:46:01.291 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100610.log.gz 2024-04-25T04:46:01.291 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103617.log 2024-04-25T04:46:01.292 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49862.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49862.log.gz 2024-04-25T04:46:01.292 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122416.log 2024-04-25T04:46:01.293 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103617.log.gz 2024-04-25T04:46:01.293 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57958.log 2024-04-25T04:46:01.294 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122416.log.gz 2024-04-25T04:46:01.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116485.log 2024-04-25T04:46:01.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57958.log.gz 2024-04-25T04:46:01.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67810.log 2024-04-25T04:46:01.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116485.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116485.log.gz 2024-04-25T04:46:01.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126453.log 2024-04-25T04:46:01.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67810.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67810.log.gz 2024-04-25T04:46:01.297 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43123.log 2024-04-25T04:46:01.298 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.126453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126453.log.gz 2024-04-25T04:46:01.298 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98678.log 2024-04-25T04:46:01.299 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43123.log.gz 2024-04-25T04:46:01.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58142.log 2024-04-25T04:46:01.300 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98678.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98678.log.gz 2024-04-25T04:46:01.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114169.log 2024-04-25T04:46:01.301 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58142.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58142.log.gz 2024-04-25T04:46:01.301 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144292.log 2024-04-25T04:46:01.302 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114169.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114169.log.gz 2024-04-25T04:46:01.302 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60798.log 2024-04-25T04:46:01.303 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144292.log.gz 2024-04-25T04:46:01.303 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69210.log 2024-04-25T04:46:01.304 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60798.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60798.log.gz 2024-04-25T04:46:01.304 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115649.log 2024-04-25T04:46:01.305 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69210.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69210.log.gz 2024-04-25T04:46:01.305 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.149823.log 2024-04-25T04:46:01.306 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115649.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115649.log.gz 2024-04-25T04:46:01.306 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80113.log 2024-04-25T04:46:01.307 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.149823.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.149823.log.gz 2024-04-25T04:46:01.307 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53101.log 2024-04-25T04:46:01.308 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80113.log.gz 2024-04-25T04:46:01.308 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107202.log 2024-04-25T04:46:01.309 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53101.log.gz 2024-04-25T04:46:01.309 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102968.log 2024-04-25T04:46:01.310 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107202.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107202.log.gz 2024-04-25T04:46:01.310 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140500.log 2024-04-25T04:46:01.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102968.log.gz 2024-04-25T04:46:01.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146455.log 2024-04-25T04:46:01.312 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140500.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140500.log.gz 2024-04-25T04:46:01.312 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128660.log 2024-04-25T04:46:01.313 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146455.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146455.log.gz 2024-04-25T04:46:01.313 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61515.log 2024-04-25T04:46:01.314 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128660.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128660.log.gz 2024-04-25T04:46:01.314 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134719.log 2024-04-25T04:46:01.315 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61515.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61515.log.gz 2024-04-25T04:46:01.315 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48366.log 2024-04-25T04:46:01.316 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134719.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134719.log.gz 2024-04-25T04:46:01.316 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51621.log 2024-04-25T04:46:01.317 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89133.log 2024-04-25T04:46:01.317 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48366.log.gz 2024-04-25T04:46:01.318 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51621.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51621.log.gz 2024-04-25T04:46:01.318 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45501.log 2024-04-25T04:46:01.319 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89133.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89133.log.gz 2024-04-25T04:46:01.319 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71189.log 2024-04-25T04:46:01.320 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45501.log.gz 2024-04-25T04:46:01.320 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45855.log 2024-04-25T04:46:01.321 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71189.log.gz 2024-04-25T04:46:01.321 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80445.log 2024-04-25T04:46:01.322 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45855.log.gz 2024-04-25T04:46:01.322 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62090.log 2024-04-25T04:46:01.323 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80445.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80445.log.gz 2024-04-25T04:46:01.323 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134438.log 2024-04-25T04:46:01.324 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62090.log.gz 2024-04-25T04:46:01.324 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100677.log 2024-04-25T04:46:01.325 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134438.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134438.log.gz 2024-04-25T04:46:01.325 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98432.log 2024-04-25T04:46:01.326 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.100677.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100677.log.gz 2024-04-25T04:46:01.326 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150111.log 2024-04-25T04:46:01.327 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98432.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98432.log.gz 2024-04-25T04:46:01.327 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65435.log 2024-04-25T04:46:01.328 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150111.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150111.log.gz 2024-04-25T04:46:01.329 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125100.log 2024-04-25T04:46:01.329 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65435.log.gz 2024-04-25T04:46:01.330 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150763.log 2024-04-25T04:46:01.330 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125100.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125100.log.gz 2024-04-25T04:46:01.330 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125919.log 2024-04-25T04:46:01.331 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150763.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150763.log.gz 2024-04-25T04:46:01.331 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159599.log 2024-04-25T04:46:01.332 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125919.log.gz 2024-04-25T04:46:01.332 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111487.log 2024-04-25T04:46:01.333 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159599.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159599.log.gz 2024-04-25T04:46:01.333 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131968.log 2024-04-25T04:46:01.334 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111487.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111487.log.gz 2024-04-25T04:46:01.334 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152344.log 2024-04-25T04:46:01.335 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131968.log.gz 2024-04-25T04:46:01.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57391.log 2024-04-25T04:46:01.336 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152344.log.gz 2024-04-25T04:46:01.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62265.log 2024-04-25T04:46:01.337 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57391.log.gz 2024-04-25T04:46:01.337 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60087.log 2024-04-25T04:46:01.338 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62265.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62265.log.gz 2024-04-25T04:46:01.338 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111914.log 2024-04-25T04:46:01.339 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60087.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60087.log.gz 2024-04-25T04:46:01.339 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59590.log 2024-04-25T04:46:01.340 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111914.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111914.log.gz 2024-04-25T04:46:01.340 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155964.log 2024-04-25T04:46:01.341 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59590.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59590.log.gz 2024-04-25T04:46:01.341 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63906.log 2024-04-25T04:46:01.342 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155964.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155964.log.gz 2024-04-25T04:46:01.342 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125651.log 2024-04-25T04:46:01.343 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63906.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63906.log.gz 2024-04-25T04:46:01.343 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60359.log 2024-04-25T04:46:01.344 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125651.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125651.log.gz 2024-04-25T04:46:01.344 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63874.log 2024-04-25T04:46:01.345 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60359.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60359.log.gz 2024-04-25T04:46:01.345 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53566.log 2024-04-25T04:46:01.346 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63874.log.gz 2024-04-25T04:46:01.346 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52365.log 2024-04-25T04:46:01.347 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53566.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53566.log.gz 2024-04-25T04:46:01.347 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66840.log 2024-04-25T04:46:01.348 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52365.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52365.log.gz 2024-04-25T04:46:01.348 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93767.log 2024-04-25T04:46:01.348 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66840.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66840.log.gz 2024-04-25T04:46:01.349 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56248.log 2024-04-25T04:46:01.349 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93767.log.gz 2024-04-25T04:46:01.349 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72912.log 2024-04-25T04:46:01.350 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56248.log.gz 2024-04-25T04:46:01.351 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105722.log 2024-04-25T04:46:01.351 INFO:teuthology.orchestra.run.smithi188.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-04-25T04:46:01.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72912.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72912.log.gz 2024-04-25T04:46:01.351 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75215.log 2024-04-25T04:46:01.352 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105722.log.gz 2024-04-25T04:46:01.353 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53668.log 2024-04-25T04:46:01.353 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75215.log.gz 2024-04-25T04:46:01.353 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77380.log 2024-04-25T04:46:01.354 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.53668.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53668.log.gz 2024-04-25T04:46:01.354 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138974.log 2024-04-25T04:46:01.355 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77380.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77380.log.gz 2024-04-25T04:46:01.355 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76222.log 2024-04-25T04:46:01.356 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138974.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138974.log.gz 2024-04-25T04:46:01.356 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152807.log 2024-04-25T04:46:01.356 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.76222.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76222.log.gz 2024-04-25T04:46:01.357 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138336.log 2024-04-25T04:46:01.357 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152807.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152807.log.gz 2024-04-25T04:46:01.358 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158927.log 2024-04-25T04:46:01.358 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138336.log.gz 2024-04-25T04:46:01.359 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126254.log 2024-04-25T04:46:01.359 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158927.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158927.log.gz 2024-04-25T04:46:01.360 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129228.log 2024-04-25T04:46:01.360 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126254.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126254.log.gz 2024-04-25T04:46:01.360 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135271.log 2024-04-25T04:46:01.361 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129228.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129228.log.gz 2024-04-25T04:46:01.361 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70931.log 2024-04-25T04:46:01.362 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135271.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135271.log.gz 2024-04-25T04:46:01.362 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129182.log 2024-04-25T04:46:01.363 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70931.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70931.log.gz 2024-04-25T04:46:01.363 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52786.log 2024-04-25T04:46:01.364 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129182.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129182.log.gz 2024-04-25T04:46:01.364 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65294.log 2024-04-25T04:46:01.365 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.52786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.52786.log.gz 2024-04-25T04:46:01.365 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68551.log 2024-04-25T04:46:01.366 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65294.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65294.log.gz 2024-04-25T04:46:01.366 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106101.log 2024-04-25T04:46:01.366 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68551.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68551.log.gz 2024-04-25T04:46:01.367 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117902.log 2024-04-25T04:46:01.367 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106101.log.gz 2024-04-25T04:46:01.368 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62786.log 2024-04-25T04:46:01.368 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117902.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117902.log.gz 2024-04-25T04:46:01.369 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145339.log 2024-04-25T04:46:01.369 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62786.log.gz 2024-04-25T04:46:01.369 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160177.log 2024-04-25T04:46:01.370 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145339.log.gz 2024-04-25T04:46:01.370 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104119.log 2024-04-25T04:46:01.371 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160177.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160177.log.gz 2024-04-25T04:46:01.371 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80657.log 2024-04-25T04:46:01.372 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104119.log.gz 2024-04-25T04:46:01.372 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150765.log 2024-04-25T04:46:01.373 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80657.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80657.log.gz 2024-04-25T04:46:01.373 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59832.log 2024-04-25T04:46:01.373 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150765.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150765.log.gz 2024-04-25T04:46:01.374 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133255.log 2024-04-25T04:46:01.375 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59832.log.gz 2024-04-25T04:46:01.375 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154149.log 2024-04-25T04:46:01.375 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133255.log.gz 2024-04-25T04:46:01.376 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72369.log 2024-04-25T04:46:01.376 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154149.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154149.log.gz 2024-04-25T04:46:01.376 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155272.log 2024-04-25T04:46:01.377 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72369.log.gz 2024-04-25T04:46:01.377 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156753.log 2024-04-25T04:46:01.378 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155272.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155272.log.gz 2024-04-25T04:46:01.378 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55062.log 2024-04-25T04:46:01.379 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156753.log.gz 2024-04-25T04:46:01.379 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151433.log 2024-04-25T04:46:01.380 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55062.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55062.log.gz 2024-04-25T04:46:01.380 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137238.log 2024-04-25T04:46:01.381 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151433.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151433.log.gz 2024-04-25T04:46:01.381 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70804.log 2024-04-25T04:46:01.382 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137238.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137238.log.gz 2024-04-25T04:46:01.382 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130699.log 2024-04-25T04:46:01.383 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70804.log.gz 2024-04-25T04:46:01.383 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106396.log 2024-04-25T04:46:01.384 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130699.log.gz 2024-04-25T04:46:01.384 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88756.log 2024-04-25T04:46:01.385 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106396.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106396.log.gz 2024-04-25T04:46:01.385 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119295.log 2024-04-25T04:46:01.385 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.88756.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88756.log.gz 2024-04-25T04:46:01.386 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131756.log 2024-04-25T04:46:01.386 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119295.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119295.log.gz 2024-04-25T04:46:01.386 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145435.log 2024-04-25T04:46:01.387 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131756.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131756.log.gz 2024-04-25T04:46:01.387 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108209.log 2024-04-25T04:46:01.388 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145435.log.gz 2024-04-25T04:46:01.388 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132236.log 2024-04-25T04:46:01.389 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108209.log.gz 2024-04-25T04:46:01.389 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121532.log 2024-04-25T04:46:01.390 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132236.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132236.log.gz 2024-04-25T04:46:01.390 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56960.log 2024-04-25T04:46:01.391 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121532.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121532.log.gz 2024-04-25T04:46:01.391 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129669.log 2024-04-25T04:46:01.391 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56960.log.gz 2024-04-25T04:46:01.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158555.log 2024-04-25T04:46:01.392 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129669.log.gz 2024-04-25T04:46:01.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154891.log 2024-04-25T04:46:01.393 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158555.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158555.log.gz 2024-04-25T04:46:01.393 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91979.log 2024-04-25T04:46:01.394 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154891.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154891.log.gz 2024-04-25T04:46:01.394 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65852.log 2024-04-25T04:46:01.395 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91979.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91979.log.gz 2024-04-25T04:46:01.395 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117754.log 2024-04-25T04:46:01.396 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.65852.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65852.log.gz 2024-04-25T04:46:01.396 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144716.log 2024-04-25T04:46:01.397 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117754.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117754.log.gz 2024-04-25T04:46:01.397 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71267.log 2024-04-25T04:46:01.398 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144716.log.gz 2024-04-25T04:46:01.398 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144953.log 2024-04-25T04:46:01.399 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71267.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71267.log.gz 2024-04-25T04:46:01.399 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130861.log 2024-04-25T04:46:01.399 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144953.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144953.log.gz 2024-04-25T04:46:01.400 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155479.log 2024-04-25T04:46:01.401 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130861.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130861.log.gz 2024-04-25T04:46:01.401 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148559.log 2024-04-25T04:46:01.401 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155479.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155479.log.gz 2024-04-25T04:46:01.402 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135028.log 2024-04-25T04:46:01.403 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148559.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148559.log.gz 2024-04-25T04:46:01.403 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75667.log 2024-04-25T04:46:01.403 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135028.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135028.log.gz 2024-04-25T04:46:01.404 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140616.log 2024-04-25T04:46:01.404 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75667.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75667.log.gz 2024-04-25T04:46:01.404 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59510.log 2024-04-25T04:46:01.405 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140616.log.gz 2024-04-25T04:46:01.405 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46720.log 2024-04-25T04:46:01.406 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59510.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59510.log.gz 2024-04-25T04:46:01.406 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80952.log 2024-04-25T04:46:01.407 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46720.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46720.log.gz 2024-04-25T04:46:01.407 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67369.log 2024-04-25T04:46:01.408 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80952.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80952.log.gz 2024-04-25T04:46:01.408 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104538.log 2024-04-25T04:46:01.409 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67369.log.gz 2024-04-25T04:46:01.409 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49501.log 2024-04-25T04:46:01.410 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104538.log.gz 2024-04-25T04:46:01.410 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136973.log 2024-04-25T04:46:01.411 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49501.log.gz 2024-04-25T04:46:01.411 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48410.log 2024-04-25T04:46:01.412 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136973.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136973.log.gz 2024-04-25T04:46:01.412 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90282.log 2024-04-25T04:46:01.413 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48410.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48410.log.gz 2024-04-25T04:46:01.413 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129959.log 2024-04-25T04:46:01.413 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90282.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90282.log.gz 2024-04-25T04:46:01.414 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126411.log 2024-04-25T04:46:01.414 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129959.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129959.log.gz 2024-04-25T04:46:01.415 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75742.log 2024-04-25T04:46:01.415 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126411.log.gz 2024-04-25T04:46:01.415 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55858.log 2024-04-25T04:46:01.416 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75742.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75742.log.gz 2024-04-25T04:46:01.416 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146457.log 2024-04-25T04:46:01.417 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55858.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55858.log.gz 2024-04-25T04:46:01.417 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136053.log 2024-04-25T04:46:01.418 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146457.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146457.log.gz 2024-04-25T04:46:01.418 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109601.log 2024-04-25T04:46:01.419 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136053.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136053.log.gz 2024-04-25T04:46:01.419 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155565.log 2024-04-25T04:46:01.420 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109601.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109601.log.gz 2024-04-25T04:46:01.420 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108698.log 2024-04-25T04:46:01.421 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155565.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155565.log.gz 2024-04-25T04:46:01.421 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41821.log 2024-04-25T04:46:01.422 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108698.log.gz 2024-04-25T04:46:01.422 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146171.log 2024-04-25T04:46:01.423 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41821.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41821.log.gz 2024-04-25T04:46:01.423 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123755.log 2024-04-25T04:46:01.424 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146171.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146171.log.gz 2024-04-25T04:46:01.424 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107132.log 2024-04-25T04:46:01.424 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.123755.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123755.log.gz 2024-04-25T04:46:01.425 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145962.log 2024-04-25T04:46:01.426 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107132.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107132.log.gz 2024-04-25T04:46:01.426 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150509.log 2024-04-25T04:46:01.426 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145962.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145962.log.gz 2024-04-25T04:46:01.427 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99632.log 2024-04-25T04:46:01.427 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150509.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150509.log.gz 2024-04-25T04:46:01.428 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55855.log 2024-04-25T04:46:01.429 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99632.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99632.log.gz 2024-04-25T04:46:01.429 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129388.log 2024-04-25T04:46:01.429 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55855.log.gz 2024-04-25T04:46:01.429 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88552.log 2024-04-25T04:46:01.430 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129388.log.gz 2024-04-25T04:46:01.430 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150972.log 2024-04-25T04:46:01.431 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88552.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88552.log.gz 2024-04-25T04:46:01.431 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141874.log 2024-04-25T04:46:01.432 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150972.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150972.log.gz 2024-04-25T04:46:01.432 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123107.log 2024-04-25T04:46:01.433 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141874.log.gz 2024-04-25T04:46:01.433 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84164.log 2024-04-25T04:46:01.434 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.123107.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123107.log.gz 2024-04-25T04:46:01.434 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158753.log 2024-04-25T04:46:01.435 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84164.log.gz 2024-04-25T04:46:01.435 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95583.log 2024-04-25T04:46:01.436 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.158753.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158753.log.gz 2024-04-25T04:46:01.436 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53294.log 2024-04-25T04:46:01.437 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95583.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95583.log.gz 2024-04-25T04:46:01.437 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120624.log 2024-04-25T04:46:01.437 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53294.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53294.log.gz 2024-04-25T04:46:01.438 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159827.log 2024-04-25T04:46:01.438 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120624.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120624.log.gz 2024-04-25T04:46:01.439 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134947.log 2024-04-25T04:46:01.439 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159827.log.gz 2024-04-25T04:46:01.439 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147820.log 2024-04-25T04:46:01.440 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134947.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134947.log.gz 2024-04-25T04:46:01.440 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82891.log 2024-04-25T04:46:01.441 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.147820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147820.log.gz 2024-04-25T04:46:01.441 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155967.log 2024-04-25T04:46:01.442 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82891.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82891.log.gz 2024-04-25T04:46:01.442 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50565.log 2024-04-25T04:46:01.443 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155967.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155967.log.gz 2024-04-25T04:46:01.443 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121354.log 2024-04-25T04:46:01.444 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50565.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50565.log.gz 2024-04-25T04:46:01.444 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44588.log 2024-04-25T04:46:01.445 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121354.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121354.log.gz 2024-04-25T04:46:01.445 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150046.log 2024-04-25T04:46:01.446 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44588.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44588.log.gz 2024-04-25T04:46:01.446 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51034.log 2024-04-25T04:46:01.447 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150046.log.gz 2024-04-25T04:46:01.447 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49841.log 2024-04-25T04:46:01.447 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51034.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51034.log.gz 2024-04-25T04:46:01.448 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64094.log 2024-04-25T04:46:01.448 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49841.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49841.log.gz 2024-04-25T04:46:01.448 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86392.log 2024-04-25T04:46:01.449 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64094.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64094.log.gz 2024-04-25T04:46:01.449 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93516.log 2024-04-25T04:46:01.450 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86392.log.gz 2024-04-25T04:46:01.450 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146616.log 2024-04-25T04:46:01.451 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120625.log 2024-04-25T04:46:01.451 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93516.log.gz 2024-04-25T04:46:01.452 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.146616.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146616.log.gz 2024-04-25T04:46:01.452 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148873.log 2024-04-25T04:46:01.453 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120625.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120625.log.gz 2024-04-25T04:46:01.453 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40608.log 2024-04-25T04:46:01.453 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148873.log.gz 2024-04-25T04:46:01.454 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79799.log 2024-04-25T04:46:01.455 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40608.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40608.log.gz 2024-04-25T04:46:01.455 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148984.log 2024-04-25T04:46:01.456 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79799.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79799.log.gz 2024-04-25T04:46:01.456 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136235.log 2024-04-25T04:46:01.457 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148984.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148984.log.gz 2024-04-25T04:46:01.457 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100239.log 2024-04-25T04:46:01.457 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136235.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136235.log.gz 2024-04-25T04:46:01.457 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63005.log 2024-04-25T04:46:01.458 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60358.log 2024-04-25T04:46:01.459 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100239.log.gz 2024-04-25T04:46:01.459 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63005.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63005.log.gz 2024-04-25T04:46:01.459 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145499.log 2024-04-25T04:46:01.460 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60358.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60358.log.gz 2024-04-25T04:46:01.460 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105055.log 2024-04-25T04:46:01.461 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145499.log.gz 2024-04-25T04:46:01.461 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133737.log 2024-04-25T04:46:01.462 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105055.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105055.log.gz 2024-04-25T04:46:01.462 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134666.log 2024-04-25T04:46:01.463 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133737.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133737.log.gz 2024-04-25T04:46:01.463 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71434.log 2024-04-25T04:46:01.464 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134666.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134666.log.gz 2024-04-25T04:46:01.464 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71393.log 2024-04-25T04:46:01.465 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71434.log.gz 2024-04-25T04:46:01.465 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70597.log 2024-04-25T04:46:01.465 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71393.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71393.log.gz 2024-04-25T04:46:01.465 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70386.log 2024-04-25T04:46:01.466 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70597.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70597.log.gz 2024-04-25T04:46:01.466 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78451.log 2024-04-25T04:46:01.467 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70386.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70386.log.gz 2024-04-25T04:46:01.467 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148095.log 2024-04-25T04:46:01.468 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78451.log.gz 2024-04-25T04:46:01.468 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71521.log 2024-04-25T04:46:01.469 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148095.log.gz 2024-04-25T04:46:01.469 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106902.log 2024-04-25T04:46:01.470 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71521.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71521.log.gz 2024-04-25T04:46:01.470 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54198.log 2024-04-25T04:46:01.470 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106902.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106902.log.gz 2024-04-25T04:46:01.471 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135160.log 2024-04-25T04:46:01.471 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54198.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54198.log.gz 2024-04-25T04:46:01.471 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85654.log 2024-04-25T04:46:01.472 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.135160.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135160.log.gz 2024-04-25T04:46:01.472 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83034.log 2024-04-25T04:46:01.473 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85654.log.gz 2024-04-25T04:46:01.473 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86490.log 2024-04-25T04:46:01.474 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83034.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83034.log.gz 2024-04-25T04:46:01.474 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61451.log 2024-04-25T04:46:01.475 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86490.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86490.log.gz 2024-04-25T04:46:01.475 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67879.log 2024-04-25T04:46:01.476 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61451.log.gz 2024-04-25T04:46:01.476 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56564.log 2024-04-25T04:46:01.477 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67879.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67879.log.gz 2024-04-25T04:46:01.477 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64095.log 2024-04-25T04:46:01.477 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56564.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56564.log.gz 2024-04-25T04:46:01.477 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138300.log 2024-04-25T04:46:01.478 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64095.log.gz 2024-04-25T04:46:01.478 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132802.log 2024-04-25T04:46:01.479 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138300.log.gz 2024-04-25T04:46:01.479 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53181.log 2024-04-25T04:46:01.480 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132802.log.gz 2024-04-25T04:46:01.480 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85817.log 2024-04-25T04:46:01.481 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53181.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53181.log.gz 2024-04-25T04:46:01.481 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98452.log 2024-04-25T04:46:01.482 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85817.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85817.log.gz 2024-04-25T04:46:01.482 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88710.log 2024-04-25T04:46:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98452.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98452.log.gz 2024-04-25T04:46:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158652.log 2024-04-25T04:46:01.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88710.log.gz 2024-04-25T04:46:01.484 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49838.log 2024-04-25T04:46:01.484 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158652.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158652.log.gz 2024-04-25T04:46:01.484 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77976.log 2024-04-25T04:46:01.485 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.49838.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49838.log.gz 2024-04-25T04:46:01.485 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46068.log 2024-04-25T04:46:01.486 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77976.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77976.log.gz 2024-04-25T04:46:01.486 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130799.log 2024-04-25T04:46:01.487 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46068.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46068.log.gz 2024-04-25T04:46:01.487 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54259.log 2024-04-25T04:46:01.488 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130799.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130799.log.gz 2024-04-25T04:46:01.488 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159281.log 2024-04-25T04:46:01.488 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54259.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54259.log.gz 2024-04-25T04:46:01.489 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143871.log 2024-04-25T04:46:01.489 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159281.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159281.log.gz 2024-04-25T04:46:01.489 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138658.log 2024-04-25T04:46:01.490 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143871.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143871.log.gz 2024-04-25T04:46:01.490 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107092.log 2024-04-25T04:46:01.491 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138658.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138658.log.gz 2024-04-25T04:46:01.491 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159786.log 2024-04-25T04:46:01.492 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107092.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107092.log.gz 2024-04-25T04:46:01.492 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54729.log 2024-04-25T04:46:01.493 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159786.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159786.log.gz 2024-04-25T04:46:01.493 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143073.log 2024-04-25T04:46:01.493 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54729.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54729.log.gz 2024-04-25T04:46:01.494 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68991.log 2024-04-25T04:46:01.494 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143073.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143073.log.gz 2024-04-25T04:46:01.494 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97013.log 2024-04-25T04:46:01.495 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68991.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68991.log.gz 2024-04-25T04:46:01.495 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148377.log 2024-04-25T04:46:01.496 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97013.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97013.log.gz 2024-04-25T04:46:01.496 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99795.log 2024-04-25T04:46:01.497 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148377.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148377.log.gz 2024-04-25T04:46:01.497 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95158.log 2024-04-25T04:46:01.498 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99795.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99795.log.gz 2024-04-25T04:46:01.498 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76725.log 2024-04-25T04:46:01.499 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95158.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95158.log.gz 2024-04-25T04:46:01.499 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48599.log 2024-04-25T04:46:01.499 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76725.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76725.log.gz 2024-04-25T04:46:01.500 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49825.log 2024-04-25T04:46:01.500 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48599.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48599.log.gz 2024-04-25T04:46:01.500 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141379.log 2024-04-25T04:46:01.501 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.49825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49825.log.gz 2024-04-25T04:46:01.501 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59550.log 2024-04-25T04:46:01.502 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141379.log.gz 2024-04-25T04:46:01.502 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126895.log 2024-04-25T04:46:01.503 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59550.log.gz 2024-04-25T04:46:01.503 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152614.log 2024-04-25T04:46:01.504 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126895.log.gz 2024-04-25T04:46:01.504 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91414.log 2024-04-25T04:46:01.504 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152614.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152614.log.gz 2024-04-25T04:46:01.505 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121387.log 2024-04-25T04:46:01.505 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91414.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91414.log.gz 2024-04-25T04:46:01.505 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95478.log 2024-04-25T04:46:01.506 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121387.log.gz 2024-04-25T04:46:01.506 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94524.log 2024-04-25T04:46:01.507 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95478.log.gz 2024-04-25T04:46:01.507 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127790.log 2024-04-25T04:46:01.508 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94524.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94524.log.gz 2024-04-25T04:46:01.508 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124724.log 2024-04-25T04:46:01.509 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127790.log.gz 2024-04-25T04:46:01.509 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125785.log 2024-04-25T04:46:01.509 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124724.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124724.log.gz 2024-04-25T04:46:01.510 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147913.log 2024-04-25T04:46:01.510 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125785.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125785.log.gz 2024-04-25T04:46:01.510 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99532.log 2024-04-25T04:46:01.511 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.147913.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147913.log.gz 2024-04-25T04:46:01.511 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83949.log 2024-04-25T04:46:01.512 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99532.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99532.log.gz 2024-04-25T04:46:01.512 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42257.log 2024-04-25T04:46:01.513 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83949.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83949.log.gz 2024-04-25T04:46:01.513 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135364.log 2024-04-25T04:46:01.514 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.42257.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42257.log.gz 2024-04-25T04:46:01.514 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147315.log 2024-04-25T04:46:01.515 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135364.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135364.log.gz 2024-04-25T04:46:01.515 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95560.log 2024-04-25T04:46:01.515 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147315.log.gz 2024-04-25T04:46:01.515 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114779.log 2024-04-25T04:46:01.516 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95560.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95560.log.gz 2024-04-25T04:46:01.516 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142601.log 2024-04-25T04:46:01.517 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.114779.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114779.log.gz 2024-04-25T04:46:01.517 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75158.log 2024-04-25T04:46:01.519 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142601.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142601.log.gz 2024-04-25T04:46:01.520 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130589.log 2024-04-25T04:46:01.520 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75158.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75158.log.gz 2024-04-25T04:46:01.520 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78017.log 2024-04-25T04:46:01.523 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130589.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130589.log.gz 2024-04-25T04:46:01.523 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74530.log 2024-04-25T04:46:01.523 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78017.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78017.log.gz 2024-04-25T04:46:01.524 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104716.log 2024-04-25T04:46:01.524 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74530.log.gz 2024-04-25T04:46:01.524 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52509.log 2024-04-25T04:46:01.524 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104716.log.gz 2024-04-25T04:46:01.525 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51348.log 2024-04-25T04:46:01.532 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52509.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52509.log.gz 2024-04-25T04:46:01.532 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147867.log 2024-04-25T04:46:01.532 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51348.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51348.log.gz 2024-04-25T04:46:01.532 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131859.log 2024-04-25T04:46:01.532 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147867.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147867.log.gz 2024-04-25T04:46:01.532 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85849.log 2024-04-25T04:46:01.533 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131859.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131859.log.gz 2024-04-25T04:46:01.533 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98453.log 2024-04-25T04:46:01.533 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85849.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85849.log.gz 2024-04-25T04:46:01.533 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116382.log 2024-04-25T04:46:01.533 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98453.log.gz 2024-04-25T04:46:01.533 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140541.log 2024-04-25T04:46:01.534 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116382.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116382.log.gz 2024-04-25T04:46:01.534 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77540.log 2024-04-25T04:46:01.534 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140541.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140541.log.gz 2024-04-25T04:46:01.534 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89334.log 2024-04-25T04:46:01.534 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77540.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77540.log.gz 2024-04-25T04:46:01.534 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147188.log 2024-04-25T04:46:01.545 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89334.log.gz 2024-04-25T04:46:01.545 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84415.log 2024-04-25T04:46:01.545 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147188.log.gz 2024-04-25T04:46:01.545 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160405.log 2024-04-25T04:46:01.545 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84415.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84415.log.gz 2024-04-25T04:46:01.545 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83395.log 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160405.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160405.log.gz 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42614.log 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83395.log.gz 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48975.log 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42614.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42614.log.gz 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93920.log 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48975.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48975.log.gz 2024-04-25T04:46:01.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128230.log 2024-04-25T04:46:01.547 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93920.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93920.log.gz 2024-04-25T04:46:01.547 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125502.log 2024-04-25T04:46:01.547 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128230.log.gz 2024-04-25T04:46:01.547 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60317.log 2024-04-25T04:46:01.547 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125502.log.gz 2024-04-25T04:46:01.547 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78095.log 2024-04-25T04:46:01.547 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60317.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60317.log.gz 2024-04-25T04:46:01.548 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101747.log 2024-04-25T04:46:01.548 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.78095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78095.log.gz 2024-04-25T04:46:01.548 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88850.log 2024-04-25T04:46:01.548 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101747.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101747.log.gz 2024-04-25T04:46:01.548 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101348.log 2024-04-25T04:46:01.548 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.88850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88850.log.gz 2024-04-25T04:46:01.548 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78856.log 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101348.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101348.log.gz 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118006.log 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78856.log.gz 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120936.log 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118006.log.gz 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45371.log 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120936.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120936.log.gz 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111433.log 2024-04-25T04:46:01.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45371.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45371.log.gz 2024-04-25T04:46:01.552 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145698.log 2024-04-25T04:46:01.552 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111433.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111433.log.gz 2024-04-25T04:46:01.552 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97653.log 2024-04-25T04:46:01.552 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145698.log.gz 2024-04-25T04:46:01.552 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71865.log 2024-04-25T04:46:01.553 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97653.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97653.log.gz 2024-04-25T04:46:01.553 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69862.log 2024-04-25T04:46:01.553 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71865.log.gz 2024-04-25T04:46:01.553 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114205.log 2024-04-25T04:46:01.554 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69862.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69862.log.gz 2024-04-25T04:46:01.554 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120071.log 2024-04-25T04:46:01.554 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114205.log.gz 2024-04-25T04:46:01.555 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85190.log 2024-04-25T04:46:01.555 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120071.log.gz 2024-04-25T04:46:01.557 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66946.log 2024-04-25T04:46:01.557 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85190.log.gz 2024-04-25T04:46:01.557 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62351.log 2024-04-25T04:46:01.558 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66946.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66946.log.gz 2024-04-25T04:46:01.558 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147020.log 2024-04-25T04:46:01.558 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62351.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62351.log.gz 2024-04-25T04:46:01.559 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118676.log 2024-04-25T04:46:01.559 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147020.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147020.log.gz 2024-04-25T04:46:01.560 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159766.log 2024-04-25T04:46:01.560 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118676.log.gz 2024-04-25T04:46:01.560 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108795.log 2024-04-25T04:46:01.561 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159766.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159766.log.gz 2024-04-25T04:46:01.561 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47820.log 2024-04-25T04:46:01.562 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108795.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108795.log.gz 2024-04-25T04:46:01.563 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90632.log 2024-04-25T04:46:01.563 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47820.log.gz 2024-04-25T04:46:01.563 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54865.log 2024-04-25T04:46:01.568 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90632.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90632.log.gz 2024-04-25T04:46:01.568 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81963.log 2024-04-25T04:46:01.569 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54865.log.gz 2024-04-25T04:46:01.569 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112307.log 2024-04-25T04:46:01.569 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81963.log.gz 2024-04-25T04:46:01.569 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82639.log 2024-04-25T04:46:01.569 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112307.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112307.log.gz 2024-04-25T04:46:01.569 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45649.log 2024-04-25T04:46:01.570 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82639.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82639.log.gz 2024-04-25T04:46:01.570 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118908.log 2024-04-25T04:46:01.577 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45649.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45649.log.gz 2024-04-25T04:46:01.577 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129249.log 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118908.log.gz 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78549.log 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129249.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129249.log.gz 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123009.log 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78549.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78549.log.gz 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.149491.log 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123009.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123009.log.gz 2024-04-25T04:46:01.578 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46155.log 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.149491.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.149491.log.gz 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88964.log 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46155.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46155.log.gz 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59834.log 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.88964.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88964.log.gz 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88474.log 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59834.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59834.log.gz 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140072.log 2024-04-25T04:46:01.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88474.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88474.log.gz 2024-04-25T04:46:01.580 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113254.log 2024-04-25T04:46:01.583 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140072.log.gz 2024-04-25T04:46:01.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81481.log 2024-04-25T04:46:01.583 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113254.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113254.log.gz 2024-04-25T04:46:01.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57492.log 2024-04-25T04:46:01.583 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81481.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81481.log.gz 2024-04-25T04:46:01.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92845.log 2024-04-25T04:46:01.584 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57492.log.gz 2024-04-25T04:46:01.584 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138379.log 2024-04-25T04:46:01.584 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92845.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92845.log.gz 2024-04-25T04:46:01.584 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58041.log 2024-04-25T04:46:01.584 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138379.log.gz 2024-04-25T04:46:01.588 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67059.log 2024-04-25T04:46:01.589 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58041.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58041.log.gz 2024-04-25T04:46:01.589 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143137.log 2024-04-25T04:46:01.589 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67059.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67059.log.gz 2024-04-25T04:46:01.589 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61154.log 2024-04-25T04:46:01.589 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143137.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143137.log.gz 2024-04-25T04:46:01.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64659.log 2024-04-25T04:46:01.590 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61154.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61154.log.gz 2024-04-25T04:46:01.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98974.log 2024-04-25T04:46:01.590 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.64659.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64659.log.gz 2024-04-25T04:46:01.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107702.log 2024-04-25T04:46:01.590 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98974.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98974.log.gz 2024-04-25T04:46:01.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92124.log 2024-04-25T04:46:01.592 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107702.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107702.log.gz 2024-04-25T04:46:01.592 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124884.log 2024-04-25T04:46:01.592 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92124.log.gz 2024-04-25T04:46:01.592 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84515.log 2024-04-25T04:46:01.593 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124884.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124884.log.gz 2024-04-25T04:46:01.593 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90232.log 2024-04-25T04:46:01.593 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84515.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84515.log.gz 2024-04-25T04:46:01.593 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106666.log 2024-04-25T04:46:01.594 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.90232.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90232.log.gz 2024-04-25T04:46:01.594 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78701.log 2024-04-25T04:46:01.594 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106666.log.gz 2024-04-25T04:46:01.594 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56708.log 2024-04-25T04:46:01.595 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78701.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78701.log.gz 2024-04-25T04:46:01.595 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125141.log 2024-04-25T04:46:01.596 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56708.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56708.log.gz 2024-04-25T04:46:01.596 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109615.log 2024-04-25T04:46:01.597 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125141.log.gz 2024-04-25T04:46:01.597 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126869.log 2024-04-25T04:46:01.597 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109615.log.gz 2024-04-25T04:46:01.598 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132281.log 2024-04-25T04:46:01.598 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126869.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126869.log.gz 2024-04-25T04:46:01.599 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158993.log 2024-04-25T04:46:01.599 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132281.log.gz 2024-04-25T04:46:01.600 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141000.log 2024-04-25T04:46:01.600 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158993.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158993.log.gz 2024-04-25T04:46:01.600 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70764.log 2024-04-25T04:46:01.601 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141000.log.gz 2024-04-25T04:46:01.601 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64680.log 2024-04-25T04:46:01.602 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70764.log.gz 2024-04-25T04:46:01.602 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107832.log 2024-04-25T04:46:01.603 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.64680.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64680.log.gz 2024-04-25T04:46:01.603 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60402.log 2024-04-25T04:46:01.604 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107832.log.gz 2024-04-25T04:46:01.604 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77426.log 2024-04-25T04:46:01.605 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60402.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60402.log.gz 2024-04-25T04:46:01.605 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73209.log 2024-04-25T04:46:01.606 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77426.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77426.log.gz 2024-04-25T04:46:01.606 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46402.log 2024-04-25T04:46:01.607 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73209.log.gz 2024-04-25T04:46:01.608 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159522.log 2024-04-25T04:46:01.608 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46402.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46402.log.gz 2024-04-25T04:46:01.608 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92768.log 2024-04-25T04:46:01.609 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159522.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159522.log.gz 2024-04-25T04:46:01.609 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59172.log 2024-04-25T04:46:01.609 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92768.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92768.log.gz 2024-04-25T04:46:01.610 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66438.log 2024-04-25T04:46:01.610 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59172.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59172.log.gz 2024-04-25T04:46:01.611 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132486.log 2024-04-25T04:46:01.611 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66438.log.gz 2024-04-25T04:46:01.612 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105639.log 2024-04-25T04:46:01.612 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132486.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132486.log.gz 2024-04-25T04:46:01.613 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145034.log 2024-04-25T04:46:01.613 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105639.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105639.log.gz 2024-04-25T04:46:01.614 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47527.log 2024-04-25T04:46:01.614 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145034.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145034.log.gz 2024-04-25T04:46:01.615 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88693.log 2024-04-25T04:46:01.615 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47527.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47527.log.gz 2024-04-25T04:46:01.615 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105198.log 2024-04-25T04:46:01.616 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88693.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88693.log.gz 2024-04-25T04:46:01.616 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128574.log 2024-04-25T04:46:01.617 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105198.log.gz 2024-04-25T04:46:01.617 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100520.log 2024-04-25T04:46:01.618 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128574.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128574.log.gz 2024-04-25T04:46:01.618 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142641.log 2024-04-25T04:46:01.619 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100520.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100520.log.gz 2024-04-25T04:46:01.619 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106250.log 2024-04-25T04:46:01.620 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142641.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142641.log.gz 2024-04-25T04:46:01.620 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74647.log 2024-04-25T04:46:01.621 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106250.log.gz 2024-04-25T04:46:01.621 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82114.log 2024-04-25T04:46:01.622 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74647.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74647.log.gz 2024-04-25T04:46:01.622 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123078.log 2024-04-25T04:46:01.622 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82114.log.gz 2024-04-25T04:46:01.623 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99911.log 2024-04-25T04:46:01.623 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.123078.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123078.log.gz 2024-04-25T04:46:01.624 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109025.log 2024-04-25T04:46:01.624 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99911.log.gz 2024-04-25T04:46:01.625 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74481.log 2024-04-25T04:46:01.625 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109025.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109025.log.gz 2024-04-25T04:46:01.626 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136416.log 2024-04-25T04:46:01.626 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74481.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74481.log.gz 2024-04-25T04:46:01.627 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66994.log 2024-04-25T04:46:01.627 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136416.log.gz 2024-04-25T04:46:01.628 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143913.log 2024-04-25T04:46:01.629 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66994.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66994.log.gz 2024-04-25T04:46:01.629 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53917.log 2024-04-25T04:46:01.629 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143913.log.gz 2024-04-25T04:46:01.630 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67411.log 2024-04-25T04:46:01.630 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53917.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53917.log.gz 2024-04-25T04:46:01.631 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93203.log 2024-04-25T04:46:01.631 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.67411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67411.log.gz 2024-04-25T04:46:01.631 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127457.log 2024-04-25T04:46:01.632 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93203.log.gz 2024-04-25T04:46:01.632 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100024.log 2024-04-25T04:46:01.633 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127457.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127457.log.gz 2024-04-25T04:46:01.633 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48870.log 2024-04-25T04:46:01.634 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100024.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100024.log.gz 2024-04-25T04:46:01.634 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116618.log 2024-04-25T04:46:01.635 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48870.log.gz 2024-04-25T04:46:01.635 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82347.log 2024-04-25T04:46:01.636 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116618.log.gz 2024-04-25T04:46:01.636 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63478.log 2024-04-25T04:46:01.637 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82347.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82347.log.gz 2024-04-25T04:46:01.637 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88712.log 2024-04-25T04:46:01.638 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63478.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63478.log.gz 2024-04-25T04:46:01.638 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68829.log 2024-04-25T04:46:01.639 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88712.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88712.log.gz 2024-04-25T04:46:01.639 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58922.log 2024-04-25T04:46:01.640 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68829.log.gz 2024-04-25T04:46:01.640 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155754.log 2024-04-25T04:46:01.641 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58922.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58922.log.gz 2024-04-25T04:46:01.641 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74733.log 2024-04-25T04:46:01.641 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155754.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155754.log.gz 2024-04-25T04:46:01.642 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120600.log 2024-04-25T04:46:01.642 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74733.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74733.log.gz 2024-04-25T04:46:01.643 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78053.log 2024-04-25T04:46:01.643 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120600.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120600.log.gz 2024-04-25T04:46:01.644 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60465.log 2024-04-25T04:46:01.645 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78053.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78053.log.gz 2024-04-25T04:46:01.645 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115289.log 2024-04-25T04:46:01.646 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60465.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60465.log.gz 2024-04-25T04:46:01.646 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159228.log 2024-04-25T04:46:01.646 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115289.log.gz 2024-04-25T04:46:01.647 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151179.log 2024-04-25T04:46:01.647 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159228.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159228.log.gz 2024-04-25T04:46:01.648 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65872.log 2024-04-25T04:46:01.649 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151179.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151179.log.gz 2024-04-25T04:46:01.649 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85272.log 2024-04-25T04:46:01.650 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65872.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65872.log.gz 2024-04-25T04:46:01.650 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124522.log 2024-04-25T04:46:01.650 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85272.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85272.log.gz 2024-04-25T04:46:01.651 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123290.log 2024-04-25T04:46:01.651 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124522.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124522.log.gz 2024-04-25T04:46:01.652 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127243.log 2024-04-25T04:46:01.652 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123290.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123290.log.gz 2024-04-25T04:46:01.653 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120957.log 2024-04-25T04:46:01.653 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127243.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127243.log.gz 2024-04-25T04:46:01.653 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74161.log 2024-04-25T04:46:01.654 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120957.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120957.log.gz 2024-04-25T04:46:01.654 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117662.log 2024-04-25T04:46:01.655 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.74161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74161.log.gz 2024-04-25T04:46:01.655 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101389.log 2024-04-25T04:46:01.656 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117662.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117662.log.gz 2024-04-25T04:46:01.657 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106644.log 2024-04-25T04:46:01.657 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101389.log.gz 2024-04-25T04:46:01.657 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79166.log 2024-04-25T04:46:01.658 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106644.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106644.log.gz 2024-04-25T04:46:01.658 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99150.log 2024-04-25T04:46:01.659 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79166.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79166.log.gz 2024-04-25T04:46:01.659 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89200.log 2024-04-25T04:46:01.661 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99150.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99150.log.gz 2024-04-25T04:46:01.661 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136436.log 2024-04-25T04:46:01.661 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89200.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89200.log.gz 2024-04-25T04:46:01.662 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139118.log 2024-04-25T04:46:01.662 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136436.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136436.log.gz 2024-04-25T04:46:01.662 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48072.log 2024-04-25T04:46:01.663 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139118.log.gz 2024-04-25T04:46:01.663 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46069.log 2024-04-25T04:46:01.664 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48072.log.gz 2024-04-25T04:46:01.664 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78963.log 2024-04-25T04:46:01.665 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46069.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46069.log.gz 2024-04-25T04:46:01.665 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88678.log 2024-04-25T04:46:01.665 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.78963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78963.log.gz 2024-04-25T04:46:01.666 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129960.log 2024-04-25T04:46:01.667 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88678.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88678.log.gz 2024-04-25T04:46:01.667 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148740.log 2024-04-25T04:46:01.667 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129960.log.gz 2024-04-25T04:46:01.668 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86797.log 2024-04-25T04:46:01.668 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148740.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148740.log.gz 2024-04-25T04:46:01.669 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127287.log 2024-04-25T04:46:01.669 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86797.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86797.log.gz 2024-04-25T04:46:01.670 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147672.log 2024-04-25T04:46:01.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127287.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127287.log.gz 2024-04-25T04:46:01.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58834.log 2024-04-25T04:46:01.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147672.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147672.log.gz 2024-04-25T04:46:01.672 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82993.log 2024-04-25T04:46:01.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58834.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58834.log.gz 2024-04-25T04:46:01.672 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155355.log 2024-04-25T04:46:01.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82993.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82993.log.gz 2024-04-25T04:46:01.673 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146231.log 2024-04-25T04:46:01.674 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155355.log.gz 2024-04-25T04:46:01.674 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133257.log 2024-04-25T04:46:01.675 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.146231.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146231.log.gz 2024-04-25T04:46:01.675 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129539.log 2024-04-25T04:46:01.676 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133257.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133257.log.gz 2024-04-25T04:46:01.676 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145342.log 2024-04-25T04:46:01.677 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129539.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129539.log.gz 2024-04-25T04:46:01.677 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144015.log 2024-04-25T04:46:01.678 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145342.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145342.log.gz 2024-04-25T04:46:01.678 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130528.log 2024-04-25T04:46:01.679 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144015.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144015.log.gz 2024-04-25T04:46:01.679 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112973.log 2024-04-25T04:46:01.680 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130528.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130528.log.gz 2024-04-25T04:46:01.680 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133890.log 2024-04-25T04:46:01.681 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112973.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112973.log.gz 2024-04-25T04:46:01.681 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80362.log 2024-04-25T04:46:01.682 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133890.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133890.log.gz 2024-04-25T04:46:01.682 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155227.log 2024-04-25T04:46:01.682 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80362.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80362.log.gz 2024-04-25T04:46:01.683 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63808.log 2024-04-25T04:46:01.684 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155227.log.gz 2024-04-25T04:46:01.684 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98608.log 2024-04-25T04:46:01.684 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.63808.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63808.log.gz 2024-04-25T04:46:01.685 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54446.log 2024-04-25T04:46:01.686 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98608.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98608.log.gz 2024-04-25T04:46:01.686 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95437.log 2024-04-25T04:46:01.687 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54446.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54446.log.gz 2024-04-25T04:46:01.687 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88923.log 2024-04-25T04:46:01.687 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95437.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95437.log.gz 2024-04-25T04:46:01.688 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104580.log 2024-04-25T04:46:01.688 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88923.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88923.log.gz 2024-04-25T04:46:01.689 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67697.log 2024-04-25T04:46:01.689 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104580.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104580.log.gz 2024-04-25T04:46:01.690 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105895.log 2024-04-25T04:46:01.690 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67697.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67697.log.gz 2024-04-25T04:46:01.691 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112494.log 2024-04-25T04:46:01.692 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105895.log.gz 2024-04-25T04:46:01.692 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97240.log 2024-04-25T04:46:01.692 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112494.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112494.log.gz 2024-04-25T04:46:01.693 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52068.log 2024-04-25T04:46:01.693 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97240.log.gz 2024-04-25T04:46:01.694 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143703.log 2024-04-25T04:46:01.694 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52068.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52068.log.gz 2024-04-25T04:46:01.694 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146273.log 2024-04-25T04:46:01.695 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143703.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143703.log.gz 2024-04-25T04:46:01.696 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147719.log 2024-04-25T04:46:01.696 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146273.log.gz 2024-04-25T04:46:01.696 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134946.log 2024-04-25T04:46:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147719.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147719.log.gz 2024-04-25T04:46:01.697 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75926.log 2024-04-25T04:46:01.698 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134946.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134946.log.gz 2024-04-25T04:46:01.698 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70448.log 2024-04-25T04:46:01.699 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75926.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75926.log.gz 2024-04-25T04:46:01.699 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53128.log 2024-04-25T04:46:01.700 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70448.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70448.log.gz 2024-04-25T04:46:01.700 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83309.log 2024-04-25T04:46:01.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53128.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53128.log.gz 2024-04-25T04:46:01.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121973.log 2024-04-25T04:46:01.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83309.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83309.log.gz 2024-04-25T04:46:01.702 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76053.log 2024-04-25T04:46:01.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121973.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121973.log.gz 2024-04-25T04:46:01.703 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146681.log 2024-04-25T04:46:01.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.76053.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76053.log.gz 2024-04-25T04:46:01.704 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125442.log 2024-04-25T04:46:01.704 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146681.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146681.log.gz 2024-04-25T04:46:01.705 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117653.log 2024-04-25T04:46:01.705 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125442.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125442.log.gz 2024-04-25T04:46:01.705 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70510.log 2024-04-25T04:46:01.706 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117653.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117653.log.gz 2024-04-25T04:46:01.706 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107958.log 2024-04-25T04:46:01.707 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70510.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70510.log.gz 2024-04-25T04:46:01.707 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85786.log 2024-04-25T04:46:01.708 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107958.log.gz 2024-04-25T04:46:01.708 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114690.log 2024-04-25T04:46:01.709 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85786.log.gz 2024-04-25T04:46:01.709 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129814.log 2024-04-25T04:46:01.710 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114690.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114690.log.gz 2024-04-25T04:46:01.710 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141092.log 2024-04-25T04:46:01.711 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129814.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129814.log.gz 2024-04-25T04:46:01.711 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110173.log 2024-04-25T04:46:01.712 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141092.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141092.log.gz 2024-04-25T04:46:01.713 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151365.log 2024-04-25T04:46:01.713 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110173.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110173.log.gz 2024-04-25T04:46:01.713 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94484.log 2024-04-25T04:46:01.714 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151365.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151365.log.gz 2024-04-25T04:46:01.714 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100724.log 2024-04-25T04:46:01.714 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94484.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94484.log.gz 2024-04-25T04:46:01.715 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120664.log 2024-04-25T04:46:01.715 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100724.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100724.log.gz 2024-04-25T04:46:01.716 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105094.log 2024-04-25T04:46:01.716 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120664.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120664.log.gz 2024-04-25T04:46:01.716 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42612.log 2024-04-25T04:46:01.717 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105094.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105094.log.gz 2024-04-25T04:46:01.718 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63074.log 2024-04-25T04:46:01.718 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42612.log.gz 2024-04-25T04:46:01.718 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52262.log 2024-04-25T04:46:01.719 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63074.log.gz 2024-04-25T04:46:01.719 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104820.log 2024-04-25T04:46:01.720 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52262.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52262.log.gz 2024-04-25T04:46:01.720 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41618.log 2024-04-25T04:46:01.721 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104820.log.gz 2024-04-25T04:46:01.721 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149937.log 2024-04-25T04:46:01.722 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.41618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41618.log.gz 2024-04-25T04:46:01.722 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95008.log 2024-04-25T04:46:01.723 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149937.log.gz 2024-04-25T04:46:01.723 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74965.log 2024-04-25T04:46:01.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95008.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95008.log.gz 2024-04-25T04:46:01.724 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51466.log 2024-04-25T04:46:01.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74965.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74965.log.gz 2024-04-25T04:46:01.725 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80216.log 2024-04-25T04:46:01.726 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51466.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51466.log.gz 2024-04-25T04:46:01.726 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86616.log 2024-04-25T04:46:01.726 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80216.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80216.log.gz 2024-04-25T04:46:01.727 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147687.log 2024-04-25T04:46:01.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86616.log.gz 2024-04-25T04:46:01.727 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142289.log 2024-04-25T04:46:01.728 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147687.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147687.log.gz 2024-04-25T04:46:01.728 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112924.log 2024-04-25T04:46:01.729 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142289.log.gz 2024-04-25T04:46:01.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80866.log 2024-04-25T04:46:01.730 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112924.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112924.log.gz 2024-04-25T04:46:01.730 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125676.log 2024-04-25T04:46:01.731 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80866.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80866.log.gz 2024-04-25T04:46:01.732 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131758.log 2024-04-25T04:46:01.732 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125676.log.gz 2024-04-25T04:46:01.732 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143241.log 2024-04-25T04:46:01.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131758.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131758.log.gz 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157791.log 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143241.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143241.log.gz 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55369.log 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157791.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157791.log.gz 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140755.log 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55369.log.gz 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126567.log 2024-04-25T04:46:01.738 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140755.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140755.log.gz 2024-04-25T04:46:01.739 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77882.log 2024-04-25T04:46:01.739 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126567.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126567.log.gz 2024-04-25T04:46:01.739 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142161.log 2024-04-25T04:46:01.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77882.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77882.log.gz 2024-04-25T04:46:01.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113616.log 2024-04-25T04:46:01.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142161.log.gz 2024-04-25T04:46:01.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76826.log 2024-04-25T04:46:01.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113616.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113616.log.gz 2024-04-25T04:46:01.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129037.log 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76826.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76826.log.gz 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160706.log 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129037.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129037.log.gz 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103828.log 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160706.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160706.log.gz 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113693.log 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103828.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103828.log.gz 2024-04-25T04:46:01.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61198.log 2024-04-25T04:46:01.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113693.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113693.log.gz 2024-04-25T04:46:01.746 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59448.log 2024-04-25T04:46:01.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61198.log.gz 2024-04-25T04:46:01.746 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130256.log 2024-04-25T04:46:01.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59448.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59448.log.gz 2024-04-25T04:46:01.746 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45014.log 2024-04-25T04:46:01.747 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130256.log.gz 2024-04-25T04:46:01.747 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142076.log 2024-04-25T04:46:01.748 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45014.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45014.log.gz 2024-04-25T04:46:01.748 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126564.log 2024-04-25T04:46:01.748 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142076.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142076.log.gz 2024-04-25T04:46:01.748 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95310.log 2024-04-25T04:46:01.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126564.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126564.log.gz 2024-04-25T04:46:01.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65583.log 2024-04-25T04:46:01.750 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95310.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95310.log.gz 2024-04-25T04:46:01.750 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76110.log 2024-04-25T04:46:01.751 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65583.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65583.log.gz 2024-04-25T04:46:01.751 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53155.log 2024-04-25T04:46:01.752 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76110.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76110.log.gz 2024-04-25T04:46:01.752 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75216.log 2024-04-25T04:46:01.752 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53155.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53155.log.gz 2024-04-25T04:46:01.753 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119613.log 2024-04-25T04:46:01.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75216.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75216.log.gz 2024-04-25T04:46:01.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62150.log 2024-04-25T04:46:01.755 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119613.log.gz 2024-04-25T04:46:01.755 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136725.log 2024-04-25T04:46:01.755 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62150.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62150.log.gz 2024-04-25T04:46:01.755 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41977.log 2024-04-25T04:46:01.756 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136725.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136725.log.gz 2024-04-25T04:46:01.756 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134720.log 2024-04-25T04:46:01.757 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.41977.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41977.log.gz 2024-04-25T04:46:01.757 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62072.log 2024-04-25T04:46:01.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134720.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134720.log.gz 2024-04-25T04:46:01.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73501.log 2024-04-25T04:46:01.759 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62072.log.gz 2024-04-25T04:46:01.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43430.log 2024-04-25T04:46:01.760 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73501.log.gz 2024-04-25T04:46:01.760 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114406.log 2024-04-25T04:46:01.761 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43430.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43430.log.gz 2024-04-25T04:46:01.761 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42430.log 2024-04-25T04:46:01.762 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114406.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114406.log.gz 2024-04-25T04:46:01.762 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62299.log 2024-04-25T04:46:01.763 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.42430.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42430.log.gz 2024-04-25T04:46:01.763 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155968.log 2024-04-25T04:46:01.763 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62299.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62299.log.gz 2024-04-25T04:46:01.764 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123604.log 2024-04-25T04:46:01.764 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155968.log.gz 2024-04-25T04:46:01.765 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99443.log 2024-04-25T04:46:01.766 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123604.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123604.log.gz 2024-04-25T04:46:01.766 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79690.log 2024-04-25T04:46:01.767 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99443.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99443.log.gz 2024-04-25T04:46:01.767 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72662.log 2024-04-25T04:46:01.767 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79690.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79690.log.gz 2024-04-25T04:46:01.768 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95157.log 2024-04-25T04:46:01.770 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72662.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72662.log.gz 2024-04-25T04:46:01.770 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54218.log 2024-04-25T04:46:01.770 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95157.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95157.log.gz 2024-04-25T04:46:01.770 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141866.log 2024-04-25T04:46:01.771 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54218.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54218.log.gz 2024-04-25T04:46:01.771 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49293.log 2024-04-25T04:46:01.773 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141866.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141866.log.gz 2024-04-25T04:46:01.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110217.log 2024-04-25T04:46:01.774 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49293.log.gz 2024-04-25T04:46:01.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57388.log 2024-04-25T04:46:01.774 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110217.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110217.log.gz 2024-04-25T04:46:01.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70385.log 2024-04-25T04:46:01.774 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57388.log.gz 2024-04-25T04:46:01.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103931.log 2024-04-25T04:46:01.777 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70385.log.gz 2024-04-25T04:46:01.777 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48031.log 2024-04-25T04:46:01.777 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103931.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103931.log.gz 2024-04-25T04:46:01.778 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160204.log 2024-04-25T04:46:01.778 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48031.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48031.log.gz 2024-04-25T04:46:01.778 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43744.log 2024-04-25T04:46:01.778 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160204.log.gz 2024-04-25T04:46:01.778 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147934.log 2024-04-25T04:46:01.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-04-25T04:46:01.780 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43744.log.gz 2024-04-25T04:46:01.781 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159740.log 2024-04-25T04:46:01.781 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.147934.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147934.log.gz 2024-04-25T04:46:01.781 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95771.log 2024-04-25T04:46:01.781 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159740.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159740.log.gz 2024-04-25T04:46:01.781 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50054.log 2024-04-25T04:46:01.781 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95771.log.gz 2024-04-25T04:46:01.781 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87895.log 2024-04-25T04:46:01.782 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.50054.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50054.log.gz 2024-04-25T04:46:01.782 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75087.log 2024-04-25T04:46:01.782 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87895.log.gz 2024-04-25T04:46:01.782 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135806.log 2024-04-25T04:46:01.783 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75087.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75087.log.gz 2024-04-25T04:46:01.783 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94485.log 2024-04-25T04:46:01.784 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135806.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135806.log.gz 2024-04-25T04:46:01.784 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75399.log 2024-04-25T04:46:01.785 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94485.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94485.log.gz 2024-04-25T04:46:01.785 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133915.log 2024-04-25T04:46:01.786 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75399.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75399.log.gz 2024-04-25T04:46:01.786 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46216.log 2024-04-25T04:46:01.787 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133915.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133915.log.gz 2024-04-25T04:46:01.787 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68717.log 2024-04-25T04:46:01.787 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46216.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46216.log.gz 2024-04-25T04:46:01.788 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160727.log 2024-04-25T04:46:01.788 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68717.log.gz 2024-04-25T04:46:01.788 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110174.log 2024-04-25T04:46:01.789 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.160727.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160727.log.gz 2024-04-25T04:46:01.789 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114231.log 2024-04-25T04:46:01.790 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110174.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110174.log.gz 2024-04-25T04:46:01.790 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98309.log 2024-04-25T04:46:01.791 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114231.log.gz 2024-04-25T04:46:01.791 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155837.log 2024-04-25T04:46:01.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98309.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98309.log.gz 2024-04-25T04:46:01.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73001.log 2024-04-25T04:46:01.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155837.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155837.log.gz 2024-04-25T04:46:01.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123202.log 2024-04-25T04:46:01.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73001.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73001.log.gz 2024-04-25T04:46:01.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100791.log 2024-04-25T04:46:01.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123202.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123202.log.gz 2024-04-25T04:46:01.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58398.log 2024-04-25T04:46:01.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.100791.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100791.log.gz 2024-04-25T04:46:01.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75206.log 2024-04-25T04:46:01.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58398.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58398.log.gz 2024-04-25T04:46:01.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85831.log 2024-04-25T04:46:01.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75206.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75206.log.gz 2024-04-25T04:46:01.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48326.log 2024-04-25T04:46:01.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85831.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85831.log.gz 2024-04-25T04:46:01.798 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43494.log 2024-04-25T04:46:01.799 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48326.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48326.log.gz 2024-04-25T04:46:01.799 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113715.log 2024-04-25T04:46:01.800 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.43494.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43494.log.gz 2024-04-25T04:46:01.800 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159560.log 2024-04-25T04:46:01.801 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113715.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113715.log.gz 2024-04-25T04:46:01.801 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59763.log 2024-04-25T04:46:01.801 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159560.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159560.log.gz 2024-04-25T04:46:01.802 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82264.log 2024-04-25T04:46:01.802 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59763.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59763.log.gz 2024-04-25T04:46:01.803 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132661.log 2024-04-25T04:46:01.803 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82264.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82264.log.gz 2024-04-25T04:46:01.804 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105216.log 2024-04-25T04:46:01.804 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132661.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132661.log.gz 2024-04-25T04:46:01.805 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41787.log 2024-04-25T04:46:01.805 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105216.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105216.log.gz 2024-04-25T04:46:01.806 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92972.log 2024-04-25T04:46:01.806 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41787.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41787.log.gz 2024-04-25T04:46:01.807 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89475.log 2024-04-25T04:46:01.807 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92972.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92972.log.gz 2024-04-25T04:46:01.807 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84205.log 2024-04-25T04:46:01.808 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89475.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89475.log.gz 2024-04-25T04:46:01.808 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55215.log 2024-04-25T04:46:01.809 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84205.log.gz 2024-04-25T04:46:01.809 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102764.log 2024-04-25T04:46:01.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55215.log.gz 2024-04-25T04:46:01.810 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152508.log 2024-04-25T04:46:01.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102764.log.gz 2024-04-25T04:46:01.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100576.log 2024-04-25T04:46:01.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152508.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152508.log.gz 2024-04-25T04:46:01.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134484.log 2024-04-25T04:46:01.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100576.log.gz 2024-04-25T04:46:01.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93604.log 2024-04-25T04:46:01.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134484.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134484.log.gz 2024-04-25T04:46:01.814 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120532.log 2024-04-25T04:46:01.814 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93604.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93604.log.gz 2024-04-25T04:46:01.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139683.log 2024-04-25T04:46:01.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120532.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120532.log.gz 2024-04-25T04:46:01.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56142.log 2024-04-25T04:46:01.816 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139683.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139683.log.gz 2024-04-25T04:46:01.817 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89863.log 2024-04-25T04:46:01.817 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56142.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56142.log.gz 2024-04-25T04:46:01.818 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105008.log 2024-04-25T04:46:01.818 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89863.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89863.log.gz 2024-04-25T04:46:01.819 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65644.log 2024-04-25T04:46:01.819 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105008.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105008.log.gz 2024-04-25T04:46:01.820 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150876.log 2024-04-25T04:46:01.820 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65644.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65644.log.gz 2024-04-25T04:46:01.820 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53287.log 2024-04-25T04:46:01.821 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150876.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150876.log.gz 2024-04-25T04:46:01.821 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42542.log 2024-04-25T04:46:01.822 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53287.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53287.log.gz 2024-04-25T04:46:01.822 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.149668.log 2024-04-25T04:46:01.823 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42542.log.gz 2024-04-25T04:46:01.823 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124194.log 2024-04-25T04:46:01.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.149668.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.149668.log.gz 2024-04-25T04:46:01.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97046.log 2024-04-25T04:46:01.825 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145742.log 2024-04-25T04:46:01.825 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124194.log.gz 2024-04-25T04:46:01.826 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97046.log.gz 2024-04-25T04:46:01.826 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64867.log 2024-04-25T04:46:01.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145742.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145742.log.gz 2024-04-25T04:46:01.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123289.log 2024-04-25T04:46:01.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64867.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64867.log.gz 2024-04-25T04:46:01.828 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92907.log 2024-04-25T04:46:01.828 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123289.log.gz 2024-04-25T04:46:01.829 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142245.log 2024-04-25T04:46:01.830 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92907.log.gz 2024-04-25T04:46:01.830 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160660.log 2024-04-25T04:46:01.831 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142245.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142245.log.gz 2024-04-25T04:46:01.831 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61008.log 2024-04-25T04:46:01.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160660.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160660.log.gz 2024-04-25T04:46:01.832 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129540.log 2024-04-25T04:46:01.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61008.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61008.log.gz 2024-04-25T04:46:01.832 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132762.log 2024-04-25T04:46:01.833 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129540.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129540.log.gz 2024-04-25T04:46:01.833 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156463.log 2024-04-25T04:46:01.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132762.log.gz 2024-04-25T04:46:01.834 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135090.log 2024-04-25T04:46:01.835 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156463.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156463.log.gz 2024-04-25T04:46:01.835 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71949.log 2024-04-25T04:46:01.836 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135090.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135090.log.gz 2024-04-25T04:46:01.836 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125047.log 2024-04-25T04:46:01.837 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71949.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71949.log.gz 2024-04-25T04:46:01.837 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121540.log 2024-04-25T04:46:01.838 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125047.log.gz 2024-04-25T04:46:01.838 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118824.log 2024-04-25T04:46:01.839 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121540.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121540.log.gz 2024-04-25T04:46:01.839 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114983.log 2024-04-25T04:46:01.840 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118824.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118824.log.gz 2024-04-25T04:46:01.840 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43250.log 2024-04-25T04:46:01.840 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114983.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114983.log.gz 2024-04-25T04:46:01.841 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144123.log 2024-04-25T04:46:01.841 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43250.log.gz 2024-04-25T04:46:01.842 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137300.log 2024-04-25T04:46:01.842 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144123.log.gz 2024-04-25T04:46:01.842 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57235.log 2024-04-25T04:46:01.843 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137300.log.gz 2024-04-25T04:46:01.843 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107152.log 2024-04-25T04:46:01.844 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57235.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57235.log.gz 2024-04-25T04:46:01.844 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113181.log 2024-04-25T04:46:01.845 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107152.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107152.log.gz 2024-04-25T04:46:01.845 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143242.log 2024-04-25T04:46:01.846 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113181.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113181.log.gz 2024-04-25T04:46:01.846 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145918.log 2024-04-25T04:46:01.846 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143242.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143242.log.gz 2024-04-25T04:46:01.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52253.log 2024-04-25T04:46:01.847 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145918.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145918.log.gz 2024-04-25T04:46:01.848 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101302.log 2024-04-25T04:46:01.848 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52253.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52253.log.gz 2024-04-25T04:46:01.849 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129324.log 2024-04-25T04:46:01.849 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101302.log.gz 2024-04-25T04:46:01.849 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92207.log 2024-04-25T04:46:01.850 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129324.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129324.log.gz 2024-04-25T04:46:01.850 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69692.log 2024-04-25T04:46:01.851 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92207.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92207.log.gz 2024-04-25T04:46:01.851 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77097.log 2024-04-25T04:46:01.852 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69692.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69692.log.gz 2024-04-25T04:46:01.852 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160014.log 2024-04-25T04:46:01.853 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77097.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77097.log.gz 2024-04-25T04:46:01.853 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108395.log 2024-04-25T04:46:01.854 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160014.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160014.log.gz 2024-04-25T04:46:01.854 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138803.log 2024-04-25T04:46:01.855 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108395.log.gz 2024-04-25T04:46:01.855 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66903.log 2024-04-25T04:46:01.855 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138803.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138803.log.gz 2024-04-25T04:46:01.856 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119039.log 2024-04-25T04:46:01.856 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66903.log.gz 2024-04-25T04:46:01.857 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129918.log 2024-04-25T04:46:01.857 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119039.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119039.log.gz 2024-04-25T04:46:01.857 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63424.log 2024-04-25T04:46:01.858 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129918.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129918.log.gz 2024-04-25T04:46:01.859 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80114.log 2024-04-25T04:46:01.859 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63424.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63424.log.gz 2024-04-25T04:46:01.859 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157417.log 2024-04-25T04:46:01.860 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80114.log.gz 2024-04-25T04:46:01.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142927.log 2024-04-25T04:46:01.861 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157417.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157417.log.gz 2024-04-25T04:46:01.861 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57024.log 2024-04-25T04:46:01.862 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142927.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142927.log.gz 2024-04-25T04:46:01.862 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108945.log 2024-04-25T04:46:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57024.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57024.log.gz 2024-04-25T04:46:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139075.log 2024-04-25T04:46:01.863 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108945.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108945.log.gz 2024-04-25T04:46:01.864 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146094.log 2024-04-25T04:46:01.865 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139075.log.gz 2024-04-25T04:46:01.865 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143364.log 2024-04-25T04:46:01.866 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146094.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146094.log.gz 2024-04-25T04:46:01.866 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85108.log 2024-04-25T04:46:01.867 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143364.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143364.log.gz 2024-04-25T04:46:01.867 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113342.log 2024-04-25T04:46:01.867 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85108.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85108.log.gz 2024-04-25T04:46:01.868 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93624.log 2024-04-25T04:46:01.868 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113342.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113342.log.gz 2024-04-25T04:46:01.868 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115227.log 2024-04-25T04:46:01.869 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93624.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93624.log.gz 2024-04-25T04:46:01.869 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87361.log 2024-04-25T04:46:01.870 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115227.log.gz 2024-04-25T04:46:01.871 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115815.log 2024-04-25T04:46:01.871 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87361.log.gz 2024-04-25T04:46:01.871 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154435.log 2024-04-25T04:46:01.872 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115815.log.gz 2024-04-25T04:46:01.873 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55217.log 2024-04-25T04:46:01.873 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154435.log.gz 2024-04-25T04:46:01.873 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95332.log 2024-04-25T04:46:01.874 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55217.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55217.log.gz 2024-04-25T04:46:01.874 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138761.log 2024-04-25T04:46:01.875 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95332.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95332.log.gz 2024-04-25T04:46:01.875 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112256.log 2024-04-25T04:46:01.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138761.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138761.log.gz 2024-04-25T04:46:01.876 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57830.log 2024-04-25T04:46:01.877 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112256.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112256.log.gz 2024-04-25T04:46:01.877 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56625.log 2024-04-25T04:46:01.878 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57830.log.gz 2024-04-25T04:46:01.878 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137274.log 2024-04-25T04:46:01.879 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56625.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56625.log.gz 2024-04-25T04:46:01.879 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120685.log 2024-04-25T04:46:01.880 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137274.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137274.log.gz 2024-04-25T04:46:01.880 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90520.log 2024-04-25T04:46:01.881 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120685.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120685.log.gz 2024-04-25T04:46:01.881 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60735.log 2024-04-25T04:46:01.882 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.90520.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90520.log.gz 2024-04-25T04:46:01.882 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70679.log 2024-04-25T04:46:01.883 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60735.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60735.log.gz 2024-04-25T04:46:01.883 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119253.log 2024-04-25T04:46:01.883 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70679.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70679.log.gz 2024-04-25T04:46:01.884 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119758.log 2024-04-25T04:46:01.884 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119253.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119253.log.gz 2024-04-25T04:46:01.885 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52960.log 2024-04-25T04:46:01.885 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119758.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119758.log.gz 2024-04-25T04:46:01.885 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48412.log 2024-04-25T04:46:01.888 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52960.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52960.log.gz 2024-04-25T04:46:01.888 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160546.log 2024-04-25T04:46:01.889 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48412.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48412.log.gz 2024-04-25T04:46:01.889 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87848.log 2024-04-25T04:46:01.889 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160546.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160546.log.gz 2024-04-25T04:46:01.889 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137164.log 2024-04-25T04:46:01.895 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.87848.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87848.log.gz 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130706.log 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137164.log.gz 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76834.log 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130706.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130706.log.gz 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67921.log 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76834.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76834.log.gz 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96071.log 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67921.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67921.log.gz 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142265.log 2024-04-25T04:46:01.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96071.log.gz 2024-04-25T04:46:01.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146095.log 2024-04-25T04:46:01.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142265.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142265.log.gz 2024-04-25T04:46:01.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83478.log 2024-04-25T04:46:01.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146095.log.gz 2024-04-25T04:46:01.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113570.log 2024-04-25T04:46:01.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83478.log.gz 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115938.log 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113570.log.gz 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91876.log 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115938.log.gz 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112446.log 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91876.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91876.log.gz 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107748.log 2024-04-25T04:46:01.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112446.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112446.log.gz 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72288.log 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107748.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107748.log.gz 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70888.log 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72288.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72288.log.gz 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116937.log 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70888.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70888.log.gz 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51401.log 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116937.log.gz 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155899.log 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51401.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51401.log.gz 2024-04-25T04:46:01.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43214.log 2024-04-25T04:46:01.907 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155899.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155899.log.gz 2024-04-25T04:46:01.907 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101730.log 2024-04-25T04:46:01.907 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43214.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43214.log.gz 2024-04-25T04:46:01.907 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144502.log 2024-04-25T04:46:01.907 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101730.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101730.log.gz 2024-04-25T04:46:01.907 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48113.log 2024-04-25T04:46:01.907 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144502.log.gz 2024-04-25T04:46:01.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50029.log 2024-04-25T04:46:01.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48113.log.gz 2024-04-25T04:46:01.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112536.log 2024-04-25T04:46:01.909 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50029.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50029.log.gz 2024-04-25T04:46:01.909 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147663.log 2024-04-25T04:46:01.910 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112536.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112536.log.gz 2024-04-25T04:46:01.910 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54426.log 2024-04-25T04:46:01.911 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147663.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147663.log.gz 2024-04-25T04:46:01.911 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105804.log 2024-04-25T04:46:01.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54426.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54426.log.gz 2024-04-25T04:46:01.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131228.log 2024-04-25T04:46:01.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105804.log.gz 2024-04-25T04:46:01.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123935.log 2024-04-25T04:46:01.913 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131228.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131228.log.gz 2024-04-25T04:46:01.913 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141360.log 2024-04-25T04:46:01.914 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123935.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123935.log.gz 2024-04-25T04:46:01.914 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155646.log 2024-04-25T04:46:01.915 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141360.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141360.log.gz 2024-04-25T04:46:01.915 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59614.log 2024-04-25T04:46:01.916 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155646.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155646.log.gz 2024-04-25T04:46:01.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101328.log 2024-04-25T04:46:01.917 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59614.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59614.log.gz 2024-04-25T04:46:01.917 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61724.log 2024-04-25T04:46:01.918 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101328.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101328.log.gz 2024-04-25T04:46:01.918 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159167.log 2024-04-25T04:46:01.918 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61724.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61724.log.gz 2024-04-25T04:46:01.918 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95650.log 2024-04-25T04:46:01.919 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159167.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159167.log.gz 2024-04-25T04:46:01.919 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57874.log 2024-04-25T04:46:01.920 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95650.log.gz 2024-04-25T04:46:01.920 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93096.log 2024-04-25T04:46:01.921 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57874.log.gz 2024-04-25T04:46:01.921 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151559.log 2024-04-25T04:46:01.922 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93096.log.gz 2024-04-25T04:46:01.922 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130909.log 2024-04-25T04:46:01.922 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151559.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151559.log.gz 2024-04-25T04:46:01.923 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106667.log 2024-04-25T04:46:01.923 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130909.log.gz 2024-04-25T04:46:01.924 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47141.log 2024-04-25T04:46:01.925 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106667.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106667.log.gz 2024-04-25T04:46:01.925 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140294.log 2024-04-25T04:46:01.925 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47141.log.gz 2024-04-25T04:46:01.926 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114167.log 2024-04-25T04:46:01.926 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140294.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140294.log.gz 2024-04-25T04:46:01.926 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87102.log 2024-04-25T04:46:01.927 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114167.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114167.log.gz 2024-04-25T04:46:01.927 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53856.log 2024-04-25T04:46:01.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.87102.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87102.log.gz 2024-04-25T04:46:01.928 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126539.log 2024-04-25T04:46:01.929 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53856.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53856.log.gz 2024-04-25T04:46:01.929 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138006.log 2024-04-25T04:46:01.929 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126539.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126539.log.gz 2024-04-25T04:46:01.929 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41943.log 2024-04-25T04:46:01.930 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138006.log.gz 2024-04-25T04:46:01.930 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48241.log 2024-04-25T04:46:01.931 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.41943.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41943.log.gz 2024-04-25T04:46:01.931 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81187.log 2024-04-25T04:46:01.932 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48241.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48241.log.gz 2024-04-25T04:46:01.932 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126226.log 2024-04-25T04:46:01.933 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81187.log.gz 2024-04-25T04:46:01.933 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71948.log 2024-04-25T04:46:01.934 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126226.log.gz 2024-04-25T04:46:01.934 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157037.log 2024-04-25T04:46:01.935 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71948.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71948.log.gz 2024-04-25T04:46:01.935 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159854.log 2024-04-25T04:46:01.936 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157037.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157037.log.gz 2024-04-25T04:46:01.936 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140711.log 2024-04-25T04:46:01.937 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159854.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159854.log.gz 2024-04-25T04:46:01.937 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117542.log 2024-04-25T04:46:01.938 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140711.log.gz 2024-04-25T04:46:01.938 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153562.log 2024-04-25T04:46:01.939 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117542.log.gz 2024-04-25T04:46:01.939 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72750.log 2024-04-25T04:46:01.939 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153562.log.gz 2024-04-25T04:46:01.940 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158654.log 2024-04-25T04:46:01.941 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72750.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72750.log.gz 2024-04-25T04:46:01.941 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114719.log 2024-04-25T04:46:01.941 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158654.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158654.log.gz 2024-04-25T04:46:01.942 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101530.log 2024-04-25T04:46:01.942 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114719.log.gz 2024-04-25T04:46:01.942 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98686.log 2024-04-25T04:46:01.943 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101530.log.gz 2024-04-25T04:46:01.943 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82071.log 2024-04-25T04:46:01.944 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98686.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98686.log.gz 2024-04-25T04:46:01.944 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63004.log 2024-04-25T04:46:01.945 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82071.log.gz 2024-04-25T04:46:01.945 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109112.log 2024-04-25T04:46:01.946 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63004.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63004.log.gz 2024-04-25T04:46:01.946 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99605.log 2024-04-25T04:46:01.947 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109112.log.gz 2024-04-25T04:46:01.947 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51017.log 2024-04-25T04:46:01.948 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99605.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99605.log.gz 2024-04-25T04:46:01.948 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125389.log 2024-04-25T04:46:01.949 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51017.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51017.log.gz 2024-04-25T04:46:01.949 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129857.log 2024-04-25T04:46:01.950 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125389.log.gz 2024-04-25T04:46:01.950 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55524.log 2024-04-25T04:46:01.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129857.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129857.log.gz 2024-04-25T04:46:01.951 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160633.log 2024-04-25T04:46:01.952 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55524.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55524.log.gz 2024-04-25T04:46:01.952 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45081.log 2024-04-25T04:46:01.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160633.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160633.log.gz 2024-04-25T04:46:01.953 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46279.log 2024-04-25T04:46:01.954 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45081.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45081.log.gz 2024-04-25T04:46:01.954 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55463.log 2024-04-25T04:46:01.955 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46279.log.gz 2024-04-25T04:46:01.955 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115188.log 2024-04-25T04:46:01.956 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55463.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55463.log.gz 2024-04-25T04:46:01.956 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69945.log 2024-04-25T04:46:01.956 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115188.log.gz 2024-04-25T04:46:01.957 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47393.log 2024-04-25T04:46:01.958 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69945.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69945.log.gz 2024-04-25T04:46:01.958 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157997.log 2024-04-25T04:46:01.959 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47393.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47393.log.gz 2024-04-25T04:46:01.959 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142031.log 2024-04-25T04:46:01.959 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157997.log.gz 2024-04-25T04:46:01.960 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151160.log 2024-04-25T04:46:01.960 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142031.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142031.log.gz 2024-04-25T04:46:01.961 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127895.log 2024-04-25T04:46:01.961 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151160.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151160.log.gz 2024-04-25T04:46:01.962 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154431.log 2024-04-25T04:46:01.962 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127895.log.gz 2024-04-25T04:46:01.963 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78365.log 2024-04-25T04:46:01.964 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154431.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154431.log.gz 2024-04-25T04:46:01.964 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84326.log 2024-04-25T04:46:01.964 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.78365.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78365.log.gz 2024-04-25T04:46:01.965 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89589.log 2024-04-25T04:46:01.967 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84326.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84326.log.gz 2024-04-25T04:46:01.967 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150329.log 2024-04-25T04:46:01.967 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89589.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89589.log.gz 2024-04-25T04:46:01.967 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113295.log 2024-04-25T04:46:01.967 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.150329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150329.log.gz 2024-04-25T04:46:01.968 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147161.log 2024-04-25T04:46:01.968 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113295.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113295.log.gz 2024-04-25T04:46:01.968 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102793.log 2024-04-25T04:46:01.969 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147161.log.gz 2024-04-25T04:46:01.969 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60820.log 2024-04-25T04:46:01.970 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102793.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102793.log.gz 2024-04-25T04:46:01.970 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62586.log 2024-04-25T04:46:01.971 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60820.log.gz 2024-04-25T04:46:01.971 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106943.log 2024-04-25T04:46:01.972 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62586.log.gz 2024-04-25T04:46:01.972 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157916.log 2024-04-25T04:46:01.973 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106943.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106943.log.gz 2024-04-25T04:46:01.973 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124960.log 2024-04-25T04:46:01.974 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157916.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157916.log.gz 2024-04-25T04:46:01.975 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154339.log 2024-04-25T04:46:01.975 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124960.log.gz 2024-04-25T04:46:01.975 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134113.log 2024-04-25T04:46:01.976 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154339.log.gz 2024-04-25T04:46:01.976 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104474.log 2024-04-25T04:46:01.977 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134113.log.gz 2024-04-25T04:46:01.977 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90366.log 2024-04-25T04:46:01.978 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104474.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104474.log.gz 2024-04-25T04:46:01.978 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64762.log 2024-04-25T04:46:01.978 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.90366.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90366.log.gz 2024-04-25T04:46:01.979 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140416.log 2024-04-25T04:46:01.980 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64762.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64762.log.gz 2024-04-25T04:46:01.980 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154558.log 2024-04-25T04:46:01.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140416.log.gz 2024-04-25T04:46:01.981 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150930.log 2024-04-25T04:46:01.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154558.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154558.log.gz 2024-04-25T04:46:01.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108000.log 2024-04-25T04:46:01.983 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150930.log.gz 2024-04-25T04:46:01.983 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118073.log 2024-04-25T04:46:01.984 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108000.log.gz 2024-04-25T04:46:01.984 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154090.log 2024-04-25T04:46:01.985 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118073.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118073.log.gz 2024-04-25T04:46:01.985 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140961.log 2024-04-25T04:46:01.985 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154090.log.gz 2024-04-25T04:46:01.986 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143956.log 2024-04-25T04:46:01.987 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140961.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140961.log.gz 2024-04-25T04:46:01.987 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151558.log 2024-04-25T04:46:01.987 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143956.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143956.log.gz 2024-04-25T04:46:01.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142559.log 2024-04-25T04:46:01.988 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151558.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151558.log.gz 2024-04-25T04:46:01.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61557.log 2024-04-25T04:46:01.989 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142559.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142559.log.gz 2024-04-25T04:46:01.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142473.log 2024-04-25T04:46:01.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61557.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61557.log.gz 2024-04-25T04:46:01.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129600.log 2024-04-25T04:46:01.991 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142473.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142473.log.gz 2024-04-25T04:46:01.991 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52594.log 2024-04-25T04:46:01.992 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129600.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129600.log.gz 2024-04-25T04:46:01.992 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.122833.log 2024-04-25T04:46:01.993 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52594.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52594.log.gz 2024-04-25T04:46:01.993 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55654.log 2024-04-25T04:46:01.994 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.122833.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.122833.log.gz 2024-04-25T04:46:01.994 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65831.log 2024-04-25T04:46:01.995 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55654.log.gz 2024-04-25T04:46:01.995 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118702.log 2024-04-25T04:46:01.996 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65831.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65831.log.gz 2024-04-25T04:46:01.996 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137830.log 2024-04-25T04:46:01.997 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118702.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118702.log.gz 2024-04-25T04:46:01.997 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140195.log 2024-04-25T04:46:01.998 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137830.log.gz 2024-04-25T04:46:01.998 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118613.log 2024-04-25T04:46:01.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140195.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140195.log.gz 2024-04-25T04:46:01.999 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109045.log 2024-04-25T04:46:01.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118613.log.gz 2024-04-25T04:46:02.000 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81227.log 2024-04-25T04:46:02.000 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109045.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109045.log.gz 2024-04-25T04:46:02.001 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152909.log 2024-04-25T04:46:02.002 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81227.log.gz 2024-04-25T04:46:02.002 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78415.log 2024-04-25T04:46:02.002 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152909.log.gz 2024-04-25T04:46:02.003 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159449.log 2024-04-25T04:46:02.003 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78415.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78415.log.gz 2024-04-25T04:46:02.003 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141444.log 2024-04-25T04:46:02.004 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159449.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159449.log.gz 2024-04-25T04:46:02.004 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128727.log 2024-04-25T04:46:02.005 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141444.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141444.log.gz 2024-04-25T04:46:02.005 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86775.log 2024-04-25T04:46:02.006 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128727.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128727.log.gz 2024-04-25T04:46:02.006 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76962.log 2024-04-25T04:46:02.007 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86775.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86775.log.gz 2024-04-25T04:46:02.007 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70596.log 2024-04-25T04:46:02.008 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76962.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76962.log.gz 2024-04-25T04:46:02.008 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100697.log 2024-04-25T04:46:02.009 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70596.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70596.log.gz 2024-04-25T04:46:02.009 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57171.log 2024-04-25T04:46:02.009 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100697.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100697.log.gz 2024-04-25T04:46:02.010 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139557.log 2024-04-25T04:46:02.010 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57171.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57171.log.gz 2024-04-25T04:46:02.011 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42478.log 2024-04-25T04:46:02.011 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139557.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139557.log.gz 2024-04-25T04:46:02.012 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69547.log 2024-04-25T04:46:02.012 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42478.log.gz 2024-04-25T04:46:02.013 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47203.log 2024-04-25T04:46:02.013 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69547.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69547.log.gz 2024-04-25T04:46:02.014 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75428.log 2024-04-25T04:46:02.014 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47203.log.gz 2024-04-25T04:46:02.015 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124348.log 2024-04-25T04:46:02.015 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75428.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75428.log.gz 2024-04-25T04:46:02.016 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126258.log 2024-04-25T04:46:02.016 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124348.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124348.log.gz 2024-04-25T04:46:02.016 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141091.log 2024-04-25T04:46:02.017 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126258.log.gz 2024-04-25T04:46:02.017 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78183.log 2024-04-25T04:46:02.018 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141091.log.gz 2024-04-25T04:46:02.018 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40754.log 2024-04-25T04:46:02.019 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78183.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78183.log.gz 2024-04-25T04:46:02.019 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71605.log 2024-04-25T04:46:02.020 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40754.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40754.log.gz 2024-04-25T04:46:02.020 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93035.log 2024-04-25T04:46:02.021 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71605.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71605.log.gz 2024-04-25T04:46:02.021 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133282.log 2024-04-25T04:46:02.022 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93035.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93035.log.gz 2024-04-25T04:46:02.022 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58923.log 2024-04-25T04:46:02.023 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133282.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133282.log.gz 2024-04-25T04:46:02.023 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141947.log 2024-04-25T04:46:02.023 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58923.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58923.log.gz 2024-04-25T04:46:02.024 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91978.log 2024-04-25T04:46:02.024 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141947.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141947.log.gz 2024-04-25T04:46:02.025 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64869.log 2024-04-25T04:46:02.026 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91978.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91978.log.gz 2024-04-25T04:46:02.026 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48684.log 2024-04-25T04:46:02.026 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64869.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64869.log.gz 2024-04-25T04:46:02.027 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132662.log 2024-04-25T04:46:02.027 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48684.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48684.log.gz 2024-04-25T04:46:02.028 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117439.log 2024-04-25T04:46:02.029 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132662.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132662.log.gz 2024-04-25T04:46:02.029 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155649.log 2024-04-25T04:46:02.029 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117439.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117439.log.gz 2024-04-25T04:46:02.030 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105894.log 2024-04-25T04:46:02.031 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155649.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155649.log.gz 2024-04-25T04:46:02.031 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106102.log 2024-04-25T04:46:02.031 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105894.log.gz 2024-04-25T04:46:02.032 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51927.log 2024-04-25T04:46:02.032 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106102.log.gz 2024-04-25T04:46:02.032 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74080.log 2024-04-25T04:46:02.033 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51927.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51927.log.gz 2024-04-25T04:46:02.033 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131000.log 2024-04-25T04:46:02.034 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74080.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74080.log.gz 2024-04-25T04:46:02.034 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96740.log 2024-04-25T04:46:02.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131000.log.gz 2024-04-25T04:46:02.035 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161215.log 2024-04-25T04:46:02.036 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96740.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96740.log.gz 2024-04-25T04:46:02.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111840.log 2024-04-25T04:46:02.038 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161215.log.gz 2024-04-25T04:46:02.038 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94738.log 2024-04-25T04:46:02.038 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111840.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111840.log.gz 2024-04-25T04:46:02.039 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149252.log 2024-04-25T04:46:02.039 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94738.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94738.log.gz 2024-04-25T04:46:02.039 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128951.log 2024-04-25T04:46:02.040 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149252.log.gz 2024-04-25T04:46:02.040 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102634.log 2024-04-25T04:46:02.040 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128951.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128951.log.gz 2024-04-25T04:46:02.041 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144416.log 2024-04-25T04:46:02.042 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102634.log.gz 2024-04-25T04:46:02.042 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143872.log 2024-04-25T04:46:02.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144416.log.gz 2024-04-25T04:46:02.043 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87574.log 2024-04-25T04:46:02.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143872.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143872.log.gz 2024-04-25T04:46:02.044 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45469.log 2024-04-25T04:46:02.044 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87574.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87574.log.gz 2024-04-25T04:46:02.045 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147726.log 2024-04-25T04:46:02.045 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45469.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45469.log.gz 2024-04-25T04:46:02.046 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113368.log 2024-04-25T04:46:02.046 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147726.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147726.log.gz 2024-04-25T04:46:02.047 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64220.log 2024-04-25T04:46:02.047 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113368.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113368.log.gz 2024-04-25T04:46:02.048 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89428.log 2024-04-25T04:46:02.048 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64220.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64220.log.gz 2024-04-25T04:46:02.048 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63182.log 2024-04-25T04:46:02.049 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89428.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89428.log.gz 2024-04-25T04:46:02.049 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58399.log 2024-04-25T04:46:02.050 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63182.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63182.log.gz 2024-04-25T04:46:02.050 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139809.log 2024-04-25T04:46:02.051 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58399.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58399.log.gz 2024-04-25T04:46:02.051 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44252.log 2024-04-25T04:46:02.052 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139809.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139809.log.gz 2024-04-25T04:46:02.052 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43892.log 2024-04-25T04:46:02.053 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44252.log.gz 2024-04-25T04:46:02.053 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139098.log 2024-04-25T04:46:02.054 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43892.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43892.log.gz 2024-04-25T04:46:02.054 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146864.log 2024-04-25T04:46:02.054 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139098.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139098.log.gz 2024-04-25T04:46:02.055 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119716.log 2024-04-25T04:46:02.056 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146864.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146864.log.gz 2024-04-25T04:46:02.056 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104246.log 2024-04-25T04:46:02.056 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119716.log.gz 2024-04-25T04:46:02.057 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129181.log 2024-04-25T04:46:02.057 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104246.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104246.log.gz 2024-04-25T04:46:02.058 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54286.log 2024-04-25T04:46:02.058 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129181.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129181.log.gz 2024-04-25T04:46:02.058 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80771.log 2024-04-25T04:46:02.059 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54286.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54286.log.gz 2024-04-25T04:46:02.059 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68824.log 2024-04-25T04:46:02.060 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80771.log.gz 2024-04-25T04:46:02.060 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81024.log 2024-04-25T04:46:02.061 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68824.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68824.log.gz 2024-04-25T04:46:02.061 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80048.log 2024-04-25T04:46:02.061 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81024.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81024.log.gz 2024-04-25T04:46:02.062 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89057.log 2024-04-25T04:46:02.063 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80048.log.gz 2024-04-25T04:46:02.063 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122623.log 2024-04-25T04:46:02.063 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89057.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89057.log.gz 2024-04-25T04:46:02.064 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140073.log 2024-04-25T04:46:02.064 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122623.log.gz 2024-04-25T04:46:02.064 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95753.log 2024-04-25T04:46:02.065 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140073.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140073.log.gz 2024-04-25T04:46:02.065 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69075.log 2024-04-25T04:46:02.066 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95753.log.gz 2024-04-25T04:46:02.066 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156457.log 2024-04-25T04:46:02.067 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69075.log.gz 2024-04-25T04:46:02.067 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118826.log 2024-04-25T04:46:02.067 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156457.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156457.log.gz 2024-04-25T04:46:02.068 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133782.log 2024-04-25T04:46:02.068 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118826.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118826.log.gz 2024-04-25T04:46:02.069 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46467.log 2024-04-25T04:46:02.069 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133782.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133782.log.gz 2024-04-25T04:46:02.069 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96110.log 2024-04-25T04:46:02.070 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46467.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46467.log.gz 2024-04-25T04:46:02.071 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99318.log 2024-04-25T04:46:02.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96110.log: 0.0%gzip -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.96110.log.gz-- 2024-04-25T04:46:02.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91666.log 2024-04-25T04:46:02.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99318.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99318.log.gz 2024-04-25T04:46:02.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138189.log 2024-04-25T04:46:02.073 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91666.log.gz 2024-04-25T04:46:02.073 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149288.log 2024-04-25T04:46:02.074 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138189.log.gz 2024-04-25T04:46:02.074 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97605.log 2024-04-25T04:46:02.075 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149288.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149288.log.gz 2024-04-25T04:46:02.076 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109314.log 2024-04-25T04:46:02.076 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97605.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97605.log.gz 2024-04-25T04:46:02.076 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136094.log 2024-04-25T04:46:02.076 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109314.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109314.log.gz 2024-04-25T04:46:02.077 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134772.log 2024-04-25T04:46:02.078 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136094.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136094.log.gz 2024-04-25T04:46:02.078 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57450.log 2024-04-25T04:46:02.078 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.134772.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134772.log.gz 2024-04-25T04:46:02.078 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73085.log 2024-04-25T04:46:02.080 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57450.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57450.log.gz 2024-04-25T04:46:02.080 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119252.log 2024-04-25T04:46:02.080 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73085.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73085.log.gz 2024-04-25T04:46:02.081 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60821.log 2024-04-25T04:46:02.081 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119252.log.gz 2024-04-25T04:46:02.082 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89407.log 2024-04-25T04:46:02.082 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60821.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60821.log.gz 2024-04-25T04:46:02.083 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152236.log 2024-04-25T04:46:02.083 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89407.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89407.log.gz 2024-04-25T04:46:02.083 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139014.log 2024-04-25T04:46:02.084 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152236.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152236.log.gz 2024-04-25T04:46:02.084 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160271.log 2024-04-25T04:46:02.085 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139014.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139014.log.gz 2024-04-25T04:46:02.085 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127960.log 2024-04-25T04:46:02.086 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.160271.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160271.log.gz 2024-04-25T04:46:02.086 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82174.log 2024-04-25T04:46:02.087 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127960.log.gz 2024-04-25T04:46:02.087 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56353.log 2024-04-25T04:46:02.088 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82174.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82174.log.gz 2024-04-25T04:46:02.089 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112502.log 2024-04-25T04:46:02.089 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56353.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56353.log.gz 2024-04-25T04:46:02.089 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41335.log 2024-04-25T04:46:02.091 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112502.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112502.log.gz 2024-04-25T04:46:02.091 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56854.log 2024-04-25T04:46:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148331.log 2024-04-25T04:46:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56854.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56854.log.gz 2024-04-25T04:46:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65180.log 2024-04-25T04:46:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148331.log.gz 2024-04-25T04:46:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65173.log 2024-04-25T04:46:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65180.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65180.log.gz 2024-04-25T04:46:02.094 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154253.log 2024-04-25T04:46:02.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65173.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65173.log.gz 2024-04-25T04:46:02.095 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112696.log 2024-04-25T04:46:02.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154253.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154253.log.gz 2024-04-25T04:46:02.097 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66173.log 2024-04-25T04:46:02.097 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112696.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112696.log.gz 2024-04-25T04:46:02.097 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95856.log 2024-04-25T04:46:02.100 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.66173.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66173.log.gz 2024-04-25T04:46:02.100 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111072.log 2024-04-25T04:46:02.100 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95856.log.gz 2024-04-25T04:46:02.100 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132027.log 2024-04-25T04:46:02.101 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111072.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111072.log.gz 2024-04-25T04:46:02.101 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140919.log 2024-04-25T04:46:02.101 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132027.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132027.log.gz 2024-04-25T04:46:02.101 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100507.log 2024-04-25T04:46:02.103 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140919.log.gz 2024-04-25T04:46:02.103 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156235.log 2024-04-25T04:46:02.104 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100507.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100507.log.gz 2024-04-25T04:46:02.104 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84056.log 2024-04-25T04:46:02.104 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156235.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156235.log.gz 2024-04-25T04:46:02.104 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126831.log 2024-04-25T04:46:02.106 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84056.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84056.log.gz 2024-04-25T04:46:02.106 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107833.log 2024-04-25T04:46:02.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126831.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126831.log.gz 2024-04-25T04:46:02.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146538.log 2024-04-25T04:46:02.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107833.log.gz 2024-04-25T04:46:02.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161068.log 2024-04-25T04:46:02.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146538.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146538.log.gz 2024-04-25T04:46:02.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140314.log 2024-04-25T04:46:02.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161068.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161068.log.gz 2024-04-25T04:46:02.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97811.log 2024-04-25T04:46:02.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140314.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140314.log.gz 2024-04-25T04:46:02.109 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73987.log 2024-04-25T04:46:02.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97811.log.gz 2024-04-25T04:46:02.110 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64546.log 2024-04-25T04:46:02.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.73987.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73987.log.gz 2024-04-25T04:46:02.111 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94232.log 2024-04-25T04:46:02.111 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64546.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64546.log.gz 2024-04-25T04:46:02.111 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74102.log 2024-04-25T04:46:02.112 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94232.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94232.log.gz 2024-04-25T04:46:02.112 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71184.log 2024-04-25T04:46:02.113 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74102.log.gz 2024-04-25T04:46:02.114 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139370.log 2024-04-25T04:46:02.114 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71184.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71184.log.gz 2024-04-25T04:46:02.114 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139619.log 2024-04-25T04:46:02.115 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139370.log.gz 2024-04-25T04:46:02.115 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106460.log 2024-04-25T04:46:02.116 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139619.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139619.log.gz 2024-04-25T04:46:02.116 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140602.log 2024-04-25T04:46:02.117 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106460.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106460.log.gz 2024-04-25T04:46:02.117 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103618.log 2024-04-25T04:46:02.118 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140602.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140602.log.gz 2024-04-25T04:46:02.118 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44208.log 2024-04-25T04:46:02.119 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103618.log.gz 2024-04-25T04:46:02.119 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58630.log 2024-04-25T04:46:02.120 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44208.log.gz 2024-04-25T04:46:02.120 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71014.log 2024-04-25T04:46:02.121 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58630.log.gz 2024-04-25T04:46:02.121 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98451.log 2024-04-25T04:46:02.122 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71014.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71014.log.gz 2024-04-25T04:46:02.122 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135182.log 2024-04-25T04:46:02.123 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98451.log.gz 2024-04-25T04:46:02.123 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131186.log 2024-04-25T04:46:02.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135182.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135182.log.gz 2024-04-25T04:46:02.124 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76153.log 2024-04-25T04:46:02.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131186.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131186.log.gz 2024-04-25T04:46:02.125 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93453.log 2024-04-25T04:46:02.125 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76153.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76153.log.gz 2024-04-25T04:46:02.127 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64873.log 2024-04-25T04:46:02.127 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93453.log.gz 2024-04-25T04:46:02.127 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126656.log 2024-04-25T04:46:02.128 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64873.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64873.log.gz 2024-04-25T04:46:02.128 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61094.log 2024-04-25T04:46:02.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126656.log.gz 2024-04-25T04:46:02.129 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63204.log 2024-04-25T04:46:02.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61094.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61094.log.gz 2024-04-25T04:46:02.130 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157615.log 2024-04-25T04:46:02.130 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63204.log.gz 2024-04-25T04:46:02.131 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117397.log 2024-04-25T04:46:02.131 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157615.log.gz 2024-04-25T04:46:02.131 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83123.log 2024-04-25T04:46:02.132 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117397.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117397.log.gz 2024-04-25T04:46:02.133 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125074.log 2024-04-25T04:46:02.133 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83123.log.gz 2024-04-25T04:46:02.133 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111654.log 2024-04-25T04:46:02.134 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125074.log.gz 2024-04-25T04:46:02.134 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112157.log 2024-04-25T04:46:02.135 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111654.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111654.log.gz 2024-04-25T04:46:02.135 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72289.log 2024-04-25T04:46:02.136 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112157.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112157.log.gz 2024-04-25T04:46:02.136 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49039.log 2024-04-25T04:46:02.137 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72289.log.gz 2024-04-25T04:46:02.137 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125006.log 2024-04-25T04:46:02.138 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49039.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49039.log.gz 2024-04-25T04:46:02.138 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101617.log 2024-04-25T04:46:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125006.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125006.log.gz 2024-04-25T04:46:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152258.log 2024-04-25T04:46:02.139 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101617.log.gz 2024-04-25T04:46:02.140 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110528.log 2024-04-25T04:46:02.141 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152258.log.gz 2024-04-25T04:46:02.141 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89220.log 2024-04-25T04:46:02.141 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110528.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110528.log.gz 2024-04-25T04:46:02.141 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80593.log 2024-04-25T04:46:02.142 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89220.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89220.log.gz 2024-04-25T04:46:02.142 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149873.log 2024-04-25T04:46:02.143 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80593.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80593.log.gz 2024-04-25T04:46:02.143 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77722.log 2024-04-25T04:46:02.144 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149873.log.gz 2024-04-25T04:46:02.144 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100230.log 2024-04-25T04:46:02.145 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77722.log.gz 2024-04-25T04:46:02.145 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44418.log 2024-04-25T04:46:02.146 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100230.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100230.log.gz 2024-04-25T04:46:02.147 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105301.log 2024-04-25T04:46:02.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44418.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44418.log.gz 2024-04-25T04:46:02.147 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71883.log 2024-04-25T04:46:02.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105301.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105301.log.gz 2024-04-25T04:46:02.148 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112509.log 2024-04-25T04:46:02.149 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71883.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71883.log.gz 2024-04-25T04:46:02.149 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153680.log 2024-04-25T04:46:02.149 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112509.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112509.log.gz 2024-04-25T04:46:02.149 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96635.log 2024-04-25T04:46:02.150 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153680.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153680.log.gz 2024-04-25T04:46:02.150 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91705.log 2024-04-25T04:46:02.151 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96635.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96635.log.gz 2024-04-25T04:46:02.151 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76313.log 2024-04-25T04:46:02.152 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91705.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91705.log.gz 2024-04-25T04:46:02.152 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86577.log 2024-04-25T04:46:02.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76313.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76313.log.gz 2024-04-25T04:46:02.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127076.log 2024-04-25T04:46:02.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86577.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86577.log.gz 2024-04-25T04:46:02.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65945.log 2024-04-25T04:46:02.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127076.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127076.log.gz 2024-04-25T04:46:02.155 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46572.log 2024-04-25T04:46:02.155 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65945.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65945.log.gz 2024-04-25T04:46:02.155 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73927.log 2024-04-25T04:46:02.156 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46572.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46572.log.gz 2024-04-25T04:46:02.157 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85402.log 2024-04-25T04:46:02.157 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73927.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73927.log.gz 2024-04-25T04:46:02.157 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71560.log 2024-04-25T04:46:02.159 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85402.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85402.log.gz 2024-04-25T04:46:02.159 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90985.log 2024-04-25T04:46:02.159 INFO:teuthology.orchestra.run.smithi188.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41335.log.gz 2024-04-25T04:46:02.159 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71560.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71560.log.gz 2024-04-25T04:46:02.159 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53315.log 2024-04-25T04:46:02.160 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.90985.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90985.log.gz 2024-04-25T04:46:02.160 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73250.log 2024-04-25T04:46:02.161 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53315.log.gz 2024-04-25T04:46:02.161 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70197.log 2024-04-25T04:46:02.165 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73250.log.gz 2024-04-25T04:46:02.166 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54145.log 2024-04-25T04:46:02.166 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70197.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70197.log.gz 2024-04-25T04:46:02.166 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63451.log 2024-04-25T04:46:02.166 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54145.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54145.log.gz 2024-04-25T04:46:02.166 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126170.log 2024-04-25T04:46:02.166 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63451.log.gz 2024-04-25T04:46:02.167 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76864.log 2024-04-25T04:46:02.167 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126170.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126170.log.gz 2024-04-25T04:46:02.167 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147316.log 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76864.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76864.log.gz 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96361.log 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147316.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147316.log.gz 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87650.log 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96361.log.gz 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136558.log 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87650.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87650.log.gz 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159900.log 2024-04-25T04:46:02.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136558.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136558.log.gz 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95090.log 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159900.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159900.log.gz 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94715.log 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95090.log.gz 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110684.log 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94715.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94715.log.gz 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119621.log 2024-04-25T04:46:02.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110684.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110684.log.gz 2024-04-25T04:46:02.175 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73912.log 2024-04-25T04:46:02.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119621.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119621.log.gz 2024-04-25T04:46:02.176 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105681.log 2024-04-25T04:46:02.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.73912.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73912.log.gz 2024-04-25T04:46:02.176 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149598.log 2024-04-25T04:46:02.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105681.log.gz 2024-04-25T04:46:02.177 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76307.log 2024-04-25T04:46:02.177 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149598.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149598.log.gz 2024-04-25T04:46:02.177 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114652.log 2024-04-25T04:46:02.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76307.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76307.log.gz 2024-04-25T04:46:02.178 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92587.log 2024-04-25T04:46:02.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114652.log.gz 2024-04-25T04:46:02.179 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72577.log 2024-04-25T04:46:02.179 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92587.log.gz 2024-04-25T04:46:02.179 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95268.log 2024-04-25T04:46:02.180 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72577.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72577.log.gz 2024-04-25T04:46:02.180 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125027.log 2024-04-25T04:46:02.181 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95268.log.gz 2024-04-25T04:46:02.181 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135575.log 2024-04-25T04:46:02.182 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125027.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125027.log.gz 2024-04-25T04:46:02.182 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77003.log 2024-04-25T04:46:02.183 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135575.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135575.log.gz 2024-04-25T04:46:02.183 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69773.log 2024-04-25T04:46:02.184 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77003.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77003.log.gz 2024-04-25T04:46:02.184 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111567.log 2024-04-25T04:46:02.184 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69773.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69773.log.gz 2024-04-25T04:46:02.185 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101503.log 2024-04-25T04:46:02.185 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111567.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111567.log.gz 2024-04-25T04:46:02.186 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117187.log 2024-04-25T04:46:02.186 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101503.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101503.log.gz 2024-04-25T04:46:02.186 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86243.log 2024-04-25T04:46:02.187 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117187.log.gz 2024-04-25T04:46:02.187 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100978.log 2024-04-25T04:46:02.188 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86243.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86243.log.gz 2024-04-25T04:46:02.188 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103826.log 2024-04-25T04:46:02.189 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100978.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100978.log.gz 2024-04-25T04:46:02.189 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126078.log 2024-04-25T04:46:02.190 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103826.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103826.log.gz 2024-04-25T04:46:02.190 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119380.log 2024-04-25T04:46:02.190 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126078.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126078.log.gz 2024-04-25T04:46:02.191 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72870.log 2024-04-25T04:46:02.191 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119380.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119380.log.gz 2024-04-25T04:46:02.192 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48825.log 2024-04-25T04:46:02.192 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72870.log.gz 2024-04-25T04:46:02.193 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90050.log 2024-04-25T04:46:02.193 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48825.log.gz 2024-04-25T04:46:02.193 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98976.log 2024-04-25T04:46:02.194 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90050.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90050.log.gz 2024-04-25T04:46:02.194 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112904.log 2024-04-25T04:46:02.195 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98976.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98976.log.gz 2024-04-25T04:46:02.195 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100237.log 2024-04-25T04:46:02.196 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112904.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112904.log.gz 2024-04-25T04:46:02.196 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110594.log 2024-04-25T04:46:02.197 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100237.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100237.log.gz 2024-04-25T04:46:02.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106181.log 2024-04-25T04:46:02.198 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110594.log.gz 2024-04-25T04:46:02.198 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42338.log 2024-04-25T04:46:02.198 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106181.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106181.log.gz 2024-04-25T04:46:02.199 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74739.log 2024-04-25T04:46:02.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42338.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42338.log.gz 2024-04-25T04:46:02.200 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134089.log 2024-04-25T04:46:02.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74739.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74739.log.gz 2024-04-25T04:46:02.200 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159536.log 2024-04-25T04:46:02.201 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134089.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134089.log.gz 2024-04-25T04:46:02.201 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155102.log 2024-04-25T04:46:02.202 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159536.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159536.log.gz 2024-04-25T04:46:02.202 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58100.log 2024-04-25T04:46:02.203 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155102.log.gz 2024-04-25T04:46:02.203 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45434.log 2024-04-25T04:46:02.204 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58100.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58100.log.gz 2024-04-25T04:46:02.204 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49164.log 2024-04-25T04:46:02.204 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45434.log.gz 2024-04-25T04:46:02.205 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72659.log 2024-04-25T04:46:02.205 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49164.log.gz 2024-04-25T04:46:02.206 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99918.log 2024-04-25T04:46:02.206 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72659.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72659.log.gz 2024-04-25T04:46:02.207 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155776.log 2024-04-25T04:46:02.207 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99918.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99918.log.gz 2024-04-25T04:46:02.207 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91560.log 2024-04-25T04:46:02.208 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155776.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155776.log.gz 2024-04-25T04:46:02.208 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128551.log 2024-04-25T04:46:02.209 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91560.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91560.log.gz 2024-04-25T04:46:02.209 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70889.log 2024-04-25T04:46:02.210 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128551.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128551.log.gz 2024-04-25T04:46:02.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154809.log 2024-04-25T04:46:02.211 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70889.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70889.log.gz 2024-04-25T04:46:02.211 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65481.log 2024-04-25T04:46:02.212 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154809.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154809.log.gz 2024-04-25T04:46:02.212 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52813.log 2024-04-25T04:46:02.212 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65481.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65481.log.gz 2024-04-25T04:46:02.213 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76842.log 2024-04-25T04:46:02.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.52813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.52813.log.gz 2024-04-25T04:46:02.214 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151285.log 2024-04-25T04:46:02.214 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76842.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76842.log.gz 2024-04-25T04:46:02.214 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95752.log 2024-04-25T04:46:02.215 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151285.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151285.log.gz 2024-04-25T04:46:02.215 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126298.log 2024-04-25T04:46:02.216 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95752.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95752.log.gz 2024-04-25T04:46:02.216 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61679.log 2024-04-25T04:46:02.217 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.126298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126298.log.gz 2024-04-25T04:46:02.217 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94932.log 2024-04-25T04:46:02.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61679.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61679.log.gz 2024-04-25T04:46:02.218 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79230.log 2024-04-25T04:46:02.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94932.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94932.log.gz 2024-04-25T04:46:02.219 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146795.log 2024-04-25T04:46:02.220 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79230.log.gz 2024-04-25T04:46:02.220 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93204.log 2024-04-25T04:46:02.220 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146795.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146795.log.gz 2024-04-25T04:46:02.221 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107369.log 2024-04-25T04:46:02.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93204.log.gz 2024-04-25T04:46:02.222 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98882.log 2024-04-25T04:46:02.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107369.log.gz 2024-04-25T04:46:02.223 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53639.log 2024-04-25T04:46:02.223 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98882.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98882.log.gz 2024-04-25T04:46:02.224 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103408.log 2024-04-25T04:46:02.224 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53639.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53639.log.gz 2024-04-25T04:46:02.225 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88027.log 2024-04-25T04:46:02.225 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103408.log.gz 2024-04-25T04:46:02.225 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64098.log 2024-04-25T04:46:02.226 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88027.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88027.log.gz 2024-04-25T04:46:02.226 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104675.log 2024-04-25T04:46:02.227 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64098.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64098.log.gz 2024-04-25T04:46:02.227 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97816.log 2024-04-25T04:46:02.228 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104675.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104675.log.gz 2024-04-25T04:46:02.228 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82513.log 2024-04-25T04:46:02.229 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97816.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97816.log.gz 2024-04-25T04:46:02.229 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119075.log 2024-04-25T04:46:02.229 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82513.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82513.log.gz 2024-04-25T04:46:02.230 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66344.log 2024-04-25T04:46:02.230 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119075.log.gz 2024-04-25T04:46:02.231 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43832.log 2024-04-25T04:46:02.231 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66344.log.gz 2024-04-25T04:46:02.231 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111906.log 2024-04-25T04:46:02.232 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43832.log.gz 2024-04-25T04:46:02.232 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150659.log 2024-04-25T04:46:02.233 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111906.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111906.log.gz 2024-04-25T04:46:02.233 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58249.log 2024-04-25T04:46:02.234 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150659.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150659.log.gz 2024-04-25T04:46:02.234 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58250.log 2024-04-25T04:46:02.235 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58249.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58249.log.gz 2024-04-25T04:46:02.235 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48598.log 2024-04-25T04:46:02.236 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58250.log.gz 2024-04-25T04:46:02.236 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158931.log 2024-04-25T04:46:02.237 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48598.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48598.log.gz 2024-04-25T04:46:02.237 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41685.log 2024-04-25T04:46:02.237 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158931.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158931.log.gz 2024-04-25T04:46:02.238 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117755.log 2024-04-25T04:46:02.238 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.41685.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41685.log.gz 2024-04-25T04:46:02.239 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106860.log 2024-04-25T04:46:02.239 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117755.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117755.log.gz 2024-04-25T04:46:02.239 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68256.log 2024-04-25T04:46:02.240 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106860.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106860.log.gz 2024-04-25T04:46:02.240 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135520.log 2024-04-25T04:46:02.241 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68256.log.gz 2024-04-25T04:46:02.241 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109423.log 2024-04-25T04:46:02.242 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135520.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135520.log.gz 2024-04-25T04:46:02.242 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67304.log 2024-04-25T04:46:02.243 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109423.log.gz 2024-04-25T04:46:02.243 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69298.log 2024-04-25T04:46:02.243 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67304.log.gz 2024-04-25T04:46:02.244 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66080.log 2024-04-25T04:46:02.244 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69298.log.gz 2024-04-25T04:46:02.245 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93394.log 2024-04-25T04:46:02.245 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66080.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66080.log.gz 2024-04-25T04:46:02.246 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.41335.log 2024-04-25T04:46:02.246 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93394.log.gz 2024-04-25T04:46:02.247 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79517.log 2024-04-25T04:46:02.247 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.0.41335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149154.log 2024-04-25T04:46:02.248 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79517.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79517.log.gz 2024-04-25T04:46:02.248 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60566.log 2024-04-25T04:46:02.249 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149154.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149154.log.gz 2024-04-25T04:46:02.249 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121267.log 2024-04-25T04:46:02.250 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60566.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60566.log.gz 2024-04-25T04:46:02.250 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114562.log 2024-04-25T04:46:02.251 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121267.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121267.log.gz 2024-04-25T04:46:02.251 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104758.log 2024-04-25T04:46:02.251 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114562.log.gz 2024-04-25T04:46:02.252 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83528.log 2024-04-25T04:46:02.252 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104758.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104758.log.gz 2024-04-25T04:46:02.252 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68802.log 2024-04-25T04:46:02.253 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83528.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83528.log.gz 2024-04-25T04:46:02.253 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96634.log 2024-04-25T04:46:02.254 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68802.log.gz 2024-04-25T04:46:02.254 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127456.log 2024-04-25T04:46:02.255 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96634.log.gz 2024-04-25T04:46:02.255 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120531.log 2024-04-25T04:46:02.255 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127456.log.gz 2024-04-25T04:46:02.256 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64810.log 2024-04-25T04:46:02.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120531.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120531.log.gz 2024-04-25T04:46:02.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56503.log 2024-04-25T04:46:02.257 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64810.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64810.log.gz 2024-04-25T04:46:02.258 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154003.log 2024-04-25T04:46:02.258 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56503.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56503.log.gz 2024-04-25T04:46:02.258 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147840.log 2024-04-25T04:46:02.259 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154003.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154003.log.gz 2024-04-25T04:46:02.259 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122223.log 2024-04-25T04:46:02.260 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147840.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147840.log.gz 2024-04-25T04:46:02.260 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132127.log 2024-04-25T04:46:02.261 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122223.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122223.log.gz 2024-04-25T04:46:02.261 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137792.log 2024-04-25T04:46:02.262 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132127.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132127.log.gz 2024-04-25T04:46:02.262 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60948.log 2024-04-25T04:46:02.262 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137792.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137792.log.gz 2024-04-25T04:46:02.263 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52586.log 2024-04-25T04:46:02.263 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60948.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60948.log.gz 2024-04-25T04:46:02.263 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77170.log 2024-04-25T04:46:02.264 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52586.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52586.log.gz 2024-04-25T04:46:02.264 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113901.log 2024-04-25T04:46:02.265 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77170.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77170.log.gz 2024-04-25T04:46:02.265 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126694.log 2024-04-25T04:46:02.266 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113901.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113901.log.gz 2024-04-25T04:46:02.266 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68300.log 2024-04-25T04:46:02.267 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126694.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126694.log.gz 2024-04-25T04:46:02.267 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49336.log 2024-04-25T04:46:02.267 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68300.log.gz 2024-04-25T04:46:02.268 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70050.log 2024-04-25T04:46:02.268 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49336.log.gz 2024-04-25T04:46:02.268 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154555.log 2024-04-25T04:46:02.269 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70050.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70050.log.gz 2024-04-25T04:46:02.269 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158846.log 2024-04-25T04:46:02.270 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154555.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154555.log.gz 2024-04-25T04:46:02.270 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64420.log 2024-04-25T04:46:02.271 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158846.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158846.log.gz 2024-04-25T04:46:02.271 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111225.log 2024-04-25T04:46:02.272 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64420.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64420.log.gz 2024-04-25T04:46:02.272 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68477.log 2024-04-25T04:46:02.272 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111225.log.gz 2024-04-25T04:46:02.273 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61093.log 2024-04-25T04:46:02.273 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68477.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68477.log.gz 2024-04-25T04:46:02.274 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65321.log 2024-04-25T04:46:02.274 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61093.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61093.log.gz 2024-04-25T04:46:02.274 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46998.log 2024-04-25T04:46:02.275 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65321.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65321.log.gz 2024-04-25T04:46:02.275 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151503.log 2024-04-25T04:46:02.276 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46998.log.gz 2024-04-25T04:46:02.276 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107960.log 2024-04-25T04:46:02.277 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151503.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151503.log.gz 2024-04-25T04:46:02.277 INFO:teuthology.orchestra.run.smithi188.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.41335.log.gz 2024-04-25T04:46:02.277 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116486.log 2024-04-25T04:46:02.278 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107960.log.gz 2024-04-25T04:46:02.278 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144333.log 2024-04-25T04:46:02.279 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116486.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116486.log.gz 2024-04-25T04:46:02.279 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56981.log 2024-04-25T04:46:02.280 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144333.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144333.log.gz 2024-04-25T04:46:02.280 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40360.log 2024-04-25T04:46:02.281 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56981.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56981.log.gz 2024-04-25T04:46:02.281 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51391.log 2024-04-25T04:46:02.282 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40360.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40360.log.gz 2024-04-25T04:46:02.282 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67229.log 2024-04-25T04:46:02.282 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51391.log.gz 2024-04-25T04:46:02.283 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90210.log 2024-04-25T04:46:02.283 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67229.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67229.log.gz 2024-04-25T04:46:02.284 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83694.log 2024-04-25T04:46:02.284 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90210.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90210.log.gz 2024-04-25T04:46:02.284 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85192.log 2024-04-25T04:46:02.285 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83694.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83694.log.gz 2024-04-25T04:46:02.285 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52190.log 2024-04-25T04:46:02.286 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85192.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85192.log.gz 2024-04-25T04:46:02.286 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98474.log 2024-04-25T04:46:02.287 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52190.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52190.log.gz 2024-04-25T04:46:02.287 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134383.log 2024-04-25T04:46:02.288 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98474.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98474.log.gz 2024-04-25T04:46:02.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115395.log 2024-04-25T04:46:02.289 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134383.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134383.log.gz 2024-04-25T04:46:02.289 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67698.log 2024-04-25T04:46:02.289 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115395.log.gz 2024-04-25T04:46:02.290 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91980.log 2024-04-25T04:46:02.290 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67698.log.gz 2024-04-25T04:46:02.291 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116128.log 2024-04-25T04:46:02.291 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91980.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91980.log.gz 2024-04-25T04:46:02.292 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116066.log 2024-04-25T04:46:02.292 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116128.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116128.log.gz 2024-04-25T04:46:02.293 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57346.log 2024-04-25T04:46:02.293 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116066.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116066.log.gz 2024-04-25T04:46:02.293 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49949.log 2024-04-25T04:46:02.294 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57346.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57346.log.gz 2024-04-25T04:46:02.294 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140582.log 2024-04-25T04:46:02.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49949.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49949.log.gz 2024-04-25T04:46:02.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112717.log 2024-04-25T04:46:02.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140582.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140582.log.gz 2024-04-25T04:46:02.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67839.log 2024-04-25T04:46:02.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.112717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112717.log.gz 2024-04-25T04:46:02.297 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81901.log 2024-04-25T04:46:02.298 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67839.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67839.log.gz 2024-04-25T04:46:02.298 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54400.log 2024-04-25T04:46:02.299 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81901.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81901.log.gz 2024-04-25T04:46:02.299 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148027.log 2024-04-25T04:46:02.299 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54400.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54400.log.gz 2024-04-25T04:46:02.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105975.log 2024-04-25T04:46:02.300 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148027.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148027.log.gz 2024-04-25T04:46:02.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116341.log 2024-04-25T04:46:02.301 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105975.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105975.log.gz 2024-04-25T04:46:02.301 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125944.log 2024-04-25T04:46:02.302 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116341.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116341.log.gz 2024-04-25T04:46:02.302 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154976.log 2024-04-25T04:46:02.303 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125944.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125944.log.gz 2024-04-25T04:46:02.303 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124408.log 2024-04-25T04:46:02.304 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154976.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154976.log.gz 2024-04-25T04:46:02.304 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46383.log 2024-04-25T04:46:02.304 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124408.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124408.log.gz 2024-04-25T04:46:02.305 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59194.log 2024-04-25T04:46:02.305 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46383.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46383.log.gz 2024-04-25T04:46:02.306 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68550.log 2024-04-25T04:46:02.306 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59194.log.gz 2024-04-25T04:46:02.306 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108334.log 2024-04-25T04:46:02.307 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68550.log.gz 2024-04-25T04:46:02.307 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127665.log 2024-04-25T04:46:02.308 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108334.log.gz 2024-04-25T04:46:02.308 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58352.log 2024-04-25T04:46:02.309 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127665.log.gz 2024-04-25T04:46:02.309 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43788.log 2024-04-25T04:46:02.309 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58352.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58352.log.gz 2024-04-25T04:46:02.310 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59511.log 2024-04-25T04:46:02.310 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43788.log.gz 2024-04-25T04:46:02.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119798.log 2024-04-25T04:46:02.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59511.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59511.log.gz 2024-04-25T04:46:02.312 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132907.log 2024-04-25T04:46:02.312 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119798.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119798.log.gz 2024-04-25T04:46:02.313 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75941.log 2024-04-25T04:46:02.313 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132907.log.gz 2024-04-25T04:46:02.313 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41838.log 2024-04-25T04:46:02.314 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75941.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75941.log.gz 2024-04-25T04:46:02.314 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84372.log 2024-04-25T04:46:02.315 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41838.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41838.log.gz 2024-04-25T04:46:02.315 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50082.log 2024-04-25T04:46:02.316 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84372.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84372.log.gz 2024-04-25T04:46:02.316 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107625.log 2024-04-25T04:46:02.316 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.50082.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50082.log.gz 2024-04-25T04:46:02.317 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103576.log 2024-04-25T04:46:02.317 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107625.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107625.log.gz 2024-04-25T04:46:02.318 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114323.log 2024-04-25T04:46:02.318 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103576.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103576.log.gz 2024-04-25T04:46:02.319 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68825.log 2024-04-25T04:46:02.319 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114323.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114323.log.gz 2024-04-25T04:46:02.320 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139225.log 2024-04-25T04:46:02.320 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68825.log.gz 2024-04-25T04:46:02.320 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136188.log 2024-04-25T04:46:02.321 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139225.log.gz 2024-04-25T04:46:02.321 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146376.log 2024-04-25T04:46:02.322 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136188.log.gz 2024-04-25T04:46:02.322 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54999.log 2024-04-25T04:46:02.323 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146376.log.gz 2024-04-25T04:46:02.323 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118109.log 2024-04-25T04:46:02.323 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54999.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54999.log.gz 2024-04-25T04:46:02.324 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111306.log 2024-04-25T04:46:02.324 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118109.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118109.log.gz 2024-04-25T04:46:02.325 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106731.log 2024-04-25T04:46:02.325 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111306.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111306.log.gz 2024-04-25T04:46:02.326 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149387.log 2024-04-25T04:46:02.326 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106731.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106731.log.gz 2024-04-25T04:46:02.326 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117020.log 2024-04-25T04:46:02.327 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149387.log.gz 2024-04-25T04:46:02.327 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139938.log 2024-04-25T04:46:02.328 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117020.log.gz 2024-04-25T04:46:02.328 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90880.log 2024-04-25T04:46:02.329 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139938.log.gz 2024-04-25T04:46:02.329 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107789.log 2024-04-25T04:46:02.329 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90880.log.gz 2024-04-25T04:46:02.330 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49245.log 2024-04-25T04:46:02.330 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107789.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107789.log.gz 2024-04-25T04:46:02.331 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147550.log 2024-04-25T04:46:02.331 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49245.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49245.log.gz 2024-04-25T04:46:02.331 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105763.log 2024-04-25T04:46:02.332 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147550.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147550.log.gz 2024-04-25T04:46:02.332 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71185.log 2024-04-25T04:46:02.333 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105763.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105763.log.gz 2024-04-25T04:46:02.333 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137408.log 2024-04-25T04:46:02.334 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71185.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71185.log.gz 2024-04-25T04:46:02.334 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66744.log 2024-04-25T04:46:02.335 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137408.log.gz 2024-04-25T04:46:02.335 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103972.log 2024-04-25T04:46:02.335 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66744.log.gz 2024-04-25T04:46:02.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129370.log 2024-04-25T04:46:02.336 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103972.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103972.log.gz 2024-04-25T04:46:02.337 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94842.log 2024-04-25T04:46:02.337 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129370.log.gz 2024-04-25T04:46:02.338 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154640.log 2024-04-25T04:46:02.338 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94842.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94842.log.gz 2024-04-25T04:46:02.338 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68781.log 2024-04-25T04:46:02.339 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154640.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154640.log.gz 2024-04-25T04:46:02.339 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46907.log 2024-04-25T04:46:02.340 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68781.log.gz 2024-04-25T04:46:02.340 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59340.log 2024-04-25T04:46:02.341 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46907.log.gz 2024-04-25T04:46:02.341 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89676.log 2024-04-25T04:46:02.342 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59340.log.gz 2024-04-25T04:46:02.342 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133601.log 2024-04-25T04:46:02.343 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89676.log.gz 2024-04-25T04:46:02.343 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107304.log 2024-04-25T04:46:02.343 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133601.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133601.log.gz 2024-04-25T04:46:02.344 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87601.log 2024-04-25T04:46:02.344 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107304.log.gz 2024-04-25T04:46:02.344 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40311.log 2024-04-25T04:46:02.345 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87601.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87601.log.gz 2024-04-25T04:46:02.345 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157837.log 2024-04-25T04:46:02.346 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40311.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40311.log.gz 2024-04-25T04:46:02.346 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56415.log 2024-04-25T04:46:02.347 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157837.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157837.log.gz 2024-04-25T04:46:02.347 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80743.log 2024-04-25T04:46:02.348 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56415.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56415.log.gz 2024-04-25T04:46:02.348 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113046.log 2024-04-25T04:46:02.349 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80743.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80743.log.gz 2024-04-25T04:46:02.349 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73419.log 2024-04-25T04:46:02.350 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113046.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113046.log.gz 2024-04-25T04:46:02.350 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113456.log 2024-04-25T04:46:02.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73419.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73419.log.gz 2024-04-25T04:46:02.351 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107562.log 2024-04-25T04:46:02.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113456.log.gz 2024-04-25T04:46:02.352 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113765.log 2024-04-25T04:46:02.352 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107562.log.gz 2024-04-25T04:46:02.353 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93371.log 2024-04-25T04:46:02.353 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113765.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113765.log.gz 2024-04-25T04:46:02.353 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84832.log 2024-04-25T04:46:02.354 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93371.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93371.log.gz 2024-04-25T04:46:02.354 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116489.log 2024-04-25T04:46:02.355 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84832.log.gz 2024-04-25T04:46:02.355 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46844.log 2024-04-25T04:46:02.356 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116489.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116489.log.gz 2024-04-25T04:46:02.356 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159328.log 2024-04-25T04:46:02.357 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46844.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46844.log.gz 2024-04-25T04:46:02.357 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150824.log 2024-04-25T04:46:02.357 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159328.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159328.log.gz 2024-04-25T04:46:02.358 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51148.log 2024-04-25T04:46:02.358 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150824.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150824.log.gz 2024-04-25T04:46:02.359 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86359.log 2024-04-25T04:46:02.359 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51148.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51148.log.gz 2024-04-25T04:46:02.360 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152194.log 2024-04-25T04:46:02.361 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86359.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86359.log.gz 2024-04-25T04:46:02.361 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68696.log 2024-04-25T04:46:02.361 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152194.log.gz 2024-04-25T04:46:02.362 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137434.log 2024-04-25T04:46:02.362 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68696.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68696.log.gz 2024-04-25T04:46:02.363 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45080.log 2024-04-25T04:46:02.363 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137434.log.gz 2024-04-25T04:46:02.363 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44586.log 2024-04-25T04:46:02.364 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45080.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45080.log.gz 2024-04-25T04:46:02.364 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58165.log 2024-04-25T04:46:02.365 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44586.log.gz 2024-04-25T04:46:02.365 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104882.log 2024-04-25T04:46:02.366 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58165.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58165.log.gz 2024-04-25T04:46:02.366 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113985.log 2024-04-25T04:46:02.367 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104882.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104882.log.gz 2024-04-25T04:46:02.367 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66574.log 2024-04-25T04:46:02.367 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113985.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113985.log.gz 2024-04-25T04:46:02.368 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137638.log 2024-04-25T04:46:02.369 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66574.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66574.log.gz 2024-04-25T04:46:02.369 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112462.log 2024-04-25T04:46:02.369 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137638.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137638.log.gz 2024-04-25T04:46:02.370 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144016.log 2024-04-25T04:46:02.370 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112462.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112462.log.gz 2024-04-25T04:46:02.371 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70618.log 2024-04-25T04:46:02.371 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144016.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144016.log.gz 2024-04-25T04:46:02.371 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81482.log 2024-04-25T04:46:02.372 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70618.log.gz 2024-04-25T04:46:02.373 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122178.log 2024-04-25T04:46:02.373 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81482.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81482.log.gz 2024-04-25T04:46:02.373 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71698.log 2024-04-25T04:46:02.374 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122178.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122178.log.gz 2024-04-25T04:46:02.374 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84475.log 2024-04-25T04:46:02.375 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71698.log.gz 2024-04-25T04:46:02.375 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107370.log 2024-04-25T04:46:02.376 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84475.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84475.log.gz 2024-04-25T04:46:02.376 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157352.log 2024-04-25T04:46:02.377 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107370.log.gz 2024-04-25T04:46:02.377 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44942.log 2024-04-25T04:46:02.377 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157352.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157352.log.gz 2024-04-25T04:46:02.378 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111098.log 2024-04-25T04:46:02.379 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44942.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44942.log.gz 2024-04-25T04:46:02.379 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98410.log 2024-04-25T04:46:02.379 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111098.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111098.log.gz 2024-04-25T04:46:02.380 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94066.log 2024-04-25T04:46:02.380 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98410.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98410.log.gz 2024-04-25T04:46:02.381 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56751.log 2024-04-25T04:46:02.381 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94066.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94066.log.gz 2024-04-25T04:46:02.382 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134330.log 2024-04-25T04:46:02.382 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56751.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56751.log.gz 2024-04-25T04:46:02.382 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72493.log 2024-04-25T04:46:02.383 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134330.log.gz 2024-04-25T04:46:02.383 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99604.log 2024-04-25T04:46:02.384 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72493.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72493.log.gz 2024-04-25T04:46:02.384 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159134.log 2024-04-25T04:46:02.385 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99604.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99604.log.gz 2024-04-25T04:46:02.385 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124193.log 2024-04-25T04:46:02.386 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159134.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159134.log.gz 2024-04-25T04:46:02.386 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127496.log 2024-04-25T04:46:02.386 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124193.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124193.log.gz 2024-04-25T04:46:02.387 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59089.log 2024-04-25T04:46:02.388 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127496.log.gz 2024-04-25T04:46:02.388 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68134.log 2024-04-25T04:46:02.388 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59089.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59089.log.gz 2024-04-25T04:46:02.389 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104366.log 2024-04-25T04:46:02.389 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68134.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68134.log.gz 2024-04-25T04:46:02.389 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60506.log 2024-04-25T04:46:02.390 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104366.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104366.log.gz 2024-04-25T04:46:02.390 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102109.log 2024-04-25T04:46:02.391 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60506.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60506.log.gz 2024-04-25T04:46:02.391 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95230.log 2024-04-25T04:46:02.392 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102109.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102109.log.gz 2024-04-25T04:46:02.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136745.log 2024-04-25T04:46:02.393 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95230.log.gz 2024-04-25T04:46:02.393 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80050.log 2024-04-25T04:46:02.394 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136745.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136745.log.gz 2024-04-25T04:46:02.394 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111357.log 2024-04-25T04:46:02.395 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80050.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80050.log.gz 2024-04-25T04:46:02.395 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152427.log 2024-04-25T04:46:02.395 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111357.log.gz 2024-04-25T04:46:02.396 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104004.log 2024-04-25T04:46:02.396 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152427.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152427.log.gz 2024-04-25T04:46:02.396 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56982.log 2024-04-25T04:46:02.397 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104004.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104004.log.gz 2024-04-25T04:46:02.397 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56039.log 2024-04-25T04:46:02.398 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56982.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56982.log.gz 2024-04-25T04:46:02.398 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44165.log 2024-04-25T04:46:02.399 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56039.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56039.log.gz 2024-04-25T04:46:02.399 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49998.log 2024-04-25T04:46:02.400 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44165.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44165.log.gz 2024-04-25T04:46:02.400 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116297.log 2024-04-25T04:46:02.401 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49998.log.gz 2024-04-25T04:46:02.401 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60883.log 2024-04-25T04:46:02.401 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116297.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116297.log.gz 2024-04-25T04:46:02.402 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142392.log 2024-04-25T04:46:02.403 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60883.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60883.log.gz 2024-04-25T04:46:02.403 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126172.log 2024-04-25T04:46:02.403 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142392.log.gz 2024-04-25T04:46:02.404 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65388.log 2024-04-25T04:46:02.404 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126172.log: 0.0%gzip -5 -- replaced with /var/log/ceph/cluster1-client.mirror.126172.log.gz--verbose -- 2024-04-25T04:46:02.404 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140293.log 2024-04-25T04:46:02.405 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.65388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65388.log.gz 2024-04-25T04:46:02.405 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84263.log 2024-04-25T04:46:02.406 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140293.log.gz 2024-04-25T04:46:02.406 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66287.log 2024-04-25T04:46:02.407 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84263.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84263.log.gz 2024-04-25T04:46:02.407 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138659.log 2024-04-25T04:46:02.407 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.66287.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66287.log.gz 2024-04-25T04:46:02.408 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92459.log 2024-04-25T04:46:02.409 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136489.log 2024-04-25T04:46:02.409 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138659.log.gz 2024-04-25T04:46:02.409 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92459.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92459.log.gz 2024-04-25T04:46:02.409 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87164.log 2024-04-25T04:46:02.410 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136489.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136489.log.gz 2024-04-25T04:46:02.410 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51569.log 2024-04-25T04:46:02.411 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87164.log.gz 2024-04-25T04:46:02.411 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57622.log 2024-04-25T04:46:02.412 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51569.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51569.log.gz 2024-04-25T04:46:02.412 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41865.log 2024-04-25T04:46:02.413 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57622.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57622.log.gz 2024-04-25T04:46:02.413 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42336.log 2024-04-25T04:46:02.414 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41865.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41865.log.gz 2024-04-25T04:46:02.414 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90452.log 2024-04-25T04:46:02.414 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42336.log.gz 2024-04-25T04:46:02.415 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85862.log 2024-04-25T04:46:02.415 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90452.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90452.log.gz 2024-04-25T04:46:02.416 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64364.log 2024-04-25T04:46:02.416 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85862.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85862.log.gz 2024-04-25T04:46:02.416 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80908.log 2024-04-25T04:46:02.417 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64364.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64364.log.gz 2024-04-25T04:46:02.417 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109656.log 2024-04-25T04:46:02.418 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80908.log.gz 2024-04-25T04:46:02.418 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55370.log 2024-04-25T04:46:02.419 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109656.log.gz 2024-04-25T04:46:02.419 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49826.log 2024-04-25T04:46:02.420 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55370.log.gz 2024-04-25T04:46:02.420 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45331.log 2024-04-25T04:46:02.421 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.49826.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49826.log.gz 2024-04-25T04:46:02.421 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94170.log 2024-04-25T04:46:02.422 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45331.log.gz 2024-04-25T04:46:02.422 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79932.log 2024-04-25T04:46:02.423 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94170.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94170.log.gz 2024-04-25T04:46:02.423 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135336.log 2024-04-25T04:46:02.423 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79932.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79932.log.gz 2024-04-25T04:46:02.424 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61473.log 2024-04-25T04:46:02.424 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135336.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135336.log.gz 2024-04-25T04:46:02.424 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69168.log 2024-04-25T04:46:02.425 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61473.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61473.log.gz 2024-04-25T04:46:02.425 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62895.log 2024-04-25T04:46:02.426 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69168.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69168.log.gz 2024-04-25T04:46:02.426 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153227.log 2024-04-25T04:46:02.427 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62895.log.gz 2024-04-25T04:46:02.427 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92417.log 2024-04-25T04:46:02.428 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153227.log.gz 2024-04-25T04:46:02.428 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126121.log 2024-04-25T04:46:02.429 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92417.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92417.log.gz 2024-04-25T04:46:02.429 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48786.log 2024-04-25T04:46:02.430 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.126121.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126121.log.gz 2024-04-25T04:46:02.430 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96843.log 2024-04-25T04:46:02.431 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48786.log.gz 2024-04-25T04:46:02.431 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60462.log 2024-04-25T04:46:02.432 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96843.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96843.log.gz 2024-04-25T04:46:02.432 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83437.log 2024-04-25T04:46:02.433 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60462.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60462.log.gz 2024-04-25T04:46:02.433 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127182.log 2024-04-25T04:46:02.433 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83437.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83437.log.gz 2024-04-25T04:46:02.434 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151644.log 2024-04-25T04:46:02.434 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127182.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127182.log.gz 2024-04-25T04:46:02.434 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76916.log 2024-04-25T04:46:02.435 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151644.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151644.log.gz 2024-04-25T04:46:02.435 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144828.log 2024-04-25T04:46:02.436 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.76916.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76916.log.gz 2024-04-25T04:46:02.436 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127850.log 2024-04-25T04:46:02.437 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144828.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144828.log.gz 2024-04-25T04:46:02.437 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73450.log 2024-04-25T04:46:02.438 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127850.log.gz 2024-04-25T04:46:02.438 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86907.log 2024-04-25T04:46:02.439 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73450.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73450.log.gz 2024-04-25T04:46:02.439 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139413.log 2024-04-25T04:46:02.440 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86907.log.gz 2024-04-25T04:46:02.440 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128381.log 2024-04-25T04:46:02.441 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139413.log.gz 2024-04-25T04:46:02.441 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108046.log 2024-04-25T04:46:02.441 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128381.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128381.log.gz 2024-04-25T04:46:02.442 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79691.log 2024-04-25T04:46:02.442 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108046.log.gz 2024-04-25T04:46:02.442 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107541.log 2024-04-25T04:46:02.443 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79691.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79691.log.gz 2024-04-25T04:46:02.443 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135926.log 2024-04-25T04:46:02.444 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107541.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107541.log.gz 2024-04-25T04:46:02.444 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102636.log 2024-04-25T04:46:02.445 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135926.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135926.log.gz 2024-04-25T04:46:02.445 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73670.log 2024-04-25T04:46:02.446 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102636.log.gz 2024-04-25T04:46:02.446 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124279.log 2024-04-25T04:46:02.446 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73670.log.gz 2024-04-25T04:46:02.447 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108905.log 2024-04-25T04:46:02.447 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124279.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124279.log.gz 2024-04-25T04:46:02.447 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129017.log 2024-04-25T04:46:02.448 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108905.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108905.log.gz 2024-04-25T04:46:02.448 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53944.log 2024-04-25T04:46:02.449 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129017.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129017.log.gz 2024-04-25T04:46:02.449 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130065.log 2024-04-25T04:46:02.450 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53944.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53944.log.gz 2024-04-25T04:46:02.450 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89179.log 2024-04-25T04:46:02.451 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130065.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130065.log.gz 2024-04-25T04:46:02.451 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136007.log 2024-04-25T04:46:02.452 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89179.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89179.log.gz 2024-04-25T04:46:02.452 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65347.log 2024-04-25T04:46:02.452 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136007.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136007.log.gz 2024-04-25T04:46:02.453 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79013.log 2024-04-25T04:46:02.453 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65347.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65347.log.gz 2024-04-25T04:46:02.454 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90570.log 2024-04-25T04:46:02.454 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79013.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79013.log.gz 2024-04-25T04:46:02.455 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70026.log 2024-04-25T04:46:02.455 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90570.log.gz 2024-04-25T04:46:02.455 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128723.log 2024-04-25T04:46:02.456 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70026.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70026.log.gz 2024-04-25T04:46:02.456 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133977.log 2024-04-25T04:46:02.457 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128723.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128723.log.gz 2024-04-25T04:46:02.457 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69548.log 2024-04-25T04:46:02.458 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133977.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133977.log.gz 2024-04-25T04:46:02.458 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114935.log 2024-04-25T04:46:02.459 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69548.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69548.log.gz 2024-04-25T04:46:02.459 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148304.log 2024-04-25T04:46:02.459 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.114935.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114935.log.gz 2024-04-25T04:46:02.460 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90761.log 2024-04-25T04:46:02.460 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148304.log.gz 2024-04-25T04:46:02.461 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113228.log 2024-04-25T04:46:02.461 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90761.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90761.log.gz 2024-04-25T04:46:02.462 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103678.log 2024-04-25T04:46:02.462 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113228.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113228.log.gz 2024-04-25T04:46:02.462 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136147.log 2024-04-25T04:46:02.463 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103678.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103678.log.gz 2024-04-25T04:46:02.463 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92438.log 2024-04-25T04:46:02.464 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136147.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136147.log.gz 2024-04-25T04:46:02.464 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158096.log 2024-04-25T04:46:02.465 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92438.log.gz 2024-04-25T04:46:02.465 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106224.log 2024-04-25T04:46:02.466 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158096.log.gz 2024-04-25T04:46:02.466 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120389.log 2024-04-25T04:46:02.466 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106224.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106224.log.gz 2024-04-25T04:46:02.467 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155606.log 2024-04-25T04:46:02.467 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120389.log.gz 2024-04-25T04:46:02.468 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148284.log 2024-04-25T04:46:02.468 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155606.log.gz 2024-04-25T04:46:02.468 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53546.log 2024-04-25T04:46:02.469 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148284.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148284.log.gz 2024-04-25T04:46:02.469 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155144.log 2024-04-25T04:46:02.470 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.53546.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53546.log.gz 2024-04-25T04:46:02.470 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50020.log 2024-04-25T04:46:02.471 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155144.log.gz 2024-04-25T04:46:02.471 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54545.log 2024-04-25T04:46:02.472 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50020.log.gz 2024-04-25T04:46:02.472 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120438.log 2024-04-25T04:46:02.472 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54545.log.gz 2024-04-25T04:46:02.473 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44757.log 2024-04-25T04:46:02.473 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120438.log.gz 2024-04-25T04:46:02.474 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53074.log 2024-04-25T04:46:02.474 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44757.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44757.log.gz 2024-04-25T04:46:02.475 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70269.log 2024-04-25T04:46:02.475 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53074.log.gz 2024-04-25T04:46:02.475 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128595.log 2024-04-25T04:46:02.476 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70269.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70269.log.gz 2024-04-25T04:46:02.476 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117230.log 2024-04-25T04:46:02.477 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128595.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128595.log.gz 2024-04-25T04:46:02.477 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68343.log 2024-04-25T04:46:02.478 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117230.log.gz 2024-04-25T04:46:02.478 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95584.log 2024-04-25T04:46:02.479 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68343.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68343.log.gz 2024-04-25T04:46:02.479 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126503.log 2024-04-25T04:46:02.480 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95584.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95584.log.gz 2024-04-25T04:46:02.480 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103262.log 2024-04-25T04:46:02.480 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126503.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126503.log.gz 2024-04-25T04:46:02.481 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115047.log 2024-04-25T04:46:02.481 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103262.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103262.log.gz 2024-04-25T04:46:02.482 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104350.log 2024-04-25T04:46:02.482 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115047.log.gz 2024-04-25T04:46:02.483 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60778.log 2024-04-25T04:46:02.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104350.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104350.log.gz 2024-04-25T04:46:02.483 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70846.log 2024-04-25T04:46:02.484 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60778.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60778.log.gz 2024-04-25T04:46:02.484 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128020.log 2024-04-25T04:46:02.485 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70846.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70846.log.gz 2024-04-25T04:46:02.485 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88733.log 2024-04-25T04:46:02.486 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128020.log.gz 2024-04-25T04:46:02.486 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45163.log 2024-04-25T04:46:02.487 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88733.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88733.log.gz 2024-04-25T04:46:02.487 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131813.log 2024-04-25T04:46:02.487 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45163.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45163.log.gz 2024-04-25T04:46:02.488 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84873.log 2024-04-25T04:46:02.488 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131813.log.gz 2024-04-25T04:46:02.489 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59833.log 2024-04-25T04:46:02.489 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84873.log.gz 2024-04-25T04:46:02.490 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53013.log 2024-04-25T04:46:02.491 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59833.log.gz 2024-04-25T04:46:02.491 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94253.log 2024-04-25T04:46:02.492 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53013.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53013.log.gz 2024-04-25T04:46:02.492 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82263.log 2024-04-25T04:46:02.493 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94253.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94253.log.gz 2024-04-25T04:46:02.493 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85854.log 2024-04-25T04:46:02.493 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82263.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82263.log.gz 2024-04-25T04:46:02.494 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74867.log 2024-04-25T04:46:02.495 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85854.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85854.log.gz 2024-04-25T04:46:02.495 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99603.log 2024-04-25T04:46:02.496 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74867.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74867.log.gz 2024-04-25T04:46:02.496 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152435.log 2024-04-25T04:46:02.496 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99603.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99603.log.gz 2024-04-25T04:46:02.497 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156496.log 2024-04-25T04:46:02.497 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152435.log.gz 2024-04-25T04:46:02.497 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157769.log 2024-04-25T04:46:02.498 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156496.log.gz 2024-04-25T04:46:02.498 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148068.log 2024-04-25T04:46:02.499 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157769.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157769.log.gz 2024-04-25T04:46:02.499 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103059.log 2024-04-25T04:46:02.500 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148068.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148068.log.gz 2024-04-25T04:46:02.500 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106271.log 2024-04-25T04:46:02.501 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103059.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103059.log.gz 2024-04-25T04:46:02.501 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63053.log 2024-04-25T04:46:02.502 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106271.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106271.log.gz 2024-04-25T04:46:02.502 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85617.log 2024-04-25T04:46:02.502 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63053.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63053.log.gz 2024-04-25T04:46:02.503 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44554.log 2024-04-25T04:46:02.503 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85617.log.gz 2024-04-25T04:46:02.504 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75616.log 2024-04-25T04:46:02.504 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44554.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44554.log.gz 2024-04-25T04:46:02.505 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103723.log 2024-04-25T04:46:02.505 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.75616.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75616.log.gz 2024-04-25T04:46:02.505 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104925.log 2024-04-25T04:46:02.506 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103723.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103723.log.gz 2024-04-25T04:46:02.506 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135292.log 2024-04-25T04:46:02.507 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104925.log.gz 2024-04-25T04:46:02.507 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137322.log 2024-04-25T04:46:02.508 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135292.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135292.log.gz 2024-04-25T04:46:02.508 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104308.log 2024-04-25T04:46:02.509 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137322.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137322.log.gz 2024-04-25T04:46:02.509 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93537.log 2024-04-25T04:46:02.510 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104308.log.gz 2024-04-25T04:46:02.510 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70723.log 2024-04-25T04:46:02.510 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46636.log 2024-04-25T04:46:02.511 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93537.log.gz 2024-04-25T04:46:02.511 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70723.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70723.log.gz 2024-04-25T04:46:02.511 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85756.log 2024-04-25T04:46:02.512 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46636.log.gz 2024-04-25T04:46:02.512 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142434.log 2024-04-25T04:46:02.513 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85756.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85756.log.gz 2024-04-25T04:46:02.513 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114384.log 2024-04-25T04:46:02.514 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142434.log.gz 2024-04-25T04:46:02.514 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91557.log 2024-04-25T04:46:02.515 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114384.log.gz 2024-04-25T04:46:02.515 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116660.log 2024-04-25T04:46:02.515 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91557.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91557.log.gz 2024-04-25T04:46:02.516 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88137.log 2024-04-25T04:46:02.516 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116660.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116660.log.gz 2024-04-25T04:46:02.516 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150048.log 2024-04-25T04:46:02.517 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88137.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88137.log.gz 2024-04-25T04:46:02.517 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155395.log 2024-04-25T04:46:02.518 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150048.log.gz 2024-04-25T04:46:02.518 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96208.log 2024-04-25T04:46:02.519 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155395.log.gz 2024-04-25T04:46:02.519 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85193.log 2024-04-25T04:46:02.520 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96208.log.gz 2024-04-25T04:46:02.520 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119800.log 2024-04-25T04:46:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85193.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85193.log.gz 2024-04-25T04:46:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56502.log 2024-04-25T04:46:02.521 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119800.log.gz 2024-04-25T04:46:02.522 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47943.log 2024-04-25T04:46:02.522 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56502.log.gz 2024-04-25T04:46:02.522 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91035.log 2024-04-25T04:46:02.523 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47943.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47943.log.gz 2024-04-25T04:46:02.523 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94588.log 2024-04-25T04:46:02.524 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91035.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91035.log.gz 2024-04-25T04:46:02.524 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160364.log 2024-04-25T04:46:02.525 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94588.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94588.log.gz 2024-04-25T04:46:02.525 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151642.log 2024-04-25T04:46:02.526 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160364.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160364.log.gz 2024-04-25T04:46:02.526 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132803.log 2024-04-25T04:46:02.527 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151642.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151642.log.gz 2024-04-25T04:46:02.527 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88142.log 2024-04-25T04:46:02.527 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132803.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132803.log.gz 2024-04-25T04:46:02.527 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46653.log 2024-04-25T04:46:02.528 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88142.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88142.log.gz 2024-04-25T04:46:02.528 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127181.log 2024-04-25T04:46:02.529 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46653.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46653.log.gz 2024-04-25T04:46:02.529 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142821.log 2024-04-25T04:46:02.530 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127181.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127181.log.gz 2024-04-25T04:46:02.530 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134437.log 2024-04-25T04:46:02.530 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142821.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142821.log.gz 2024-04-25T04:46:02.531 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148048.log 2024-04-25T04:46:02.532 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134437.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134437.log.gz 2024-04-25T04:46:02.532 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152679.log 2024-04-25T04:46:02.533 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148048.log.gz 2024-04-25T04:46:02.533 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131419.log 2024-04-25T04:46:02.534 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152679.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152679.log.gz 2024-04-25T04:46:02.534 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104966.log 2024-04-25T04:46:02.535 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131419.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131419.log.gz 2024-04-25T04:46:02.535 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57126.log 2024-04-25T04:46:02.536 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104966.log.gz 2024-04-25T04:46:02.536 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83993.log 2024-04-25T04:46:02.536 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57126.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57126.log.gz 2024-04-25T04:46:02.536 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93141.log 2024-04-25T04:46:02.537 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83993.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83993.log.gz 2024-04-25T04:46:02.537 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104922.log 2024-04-25T04:46:02.538 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93141.log.gz 2024-04-25T04:46:02.538 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140670.log 2024-04-25T04:46:02.539 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104922.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104922.log.gz 2024-04-25T04:46:02.539 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82010.log 2024-04-25T04:46:02.539 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140670.log.gz 2024-04-25T04:46:02.540 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60403.log 2024-04-25T04:46:02.540 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82010.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82010.log.gz 2024-04-25T04:46:02.540 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93685.log 2024-04-25T04:46:02.541 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60403.log.gz 2024-04-25T04:46:02.541 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99187.log 2024-04-25T04:46:02.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93685.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93685.log.gz 2024-04-25T04:46:02.542 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47391.log 2024-04-25T04:46:02.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99187.log.gz 2024-04-25T04:46:02.543 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59802.log 2024-04-25T04:46:02.544 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47391.log.gz 2024-04-25T04:46:02.544 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153563.log 2024-04-25T04:46:02.545 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59802.log.gz 2024-04-25T04:46:02.545 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118950.log 2024-04-25T04:46:02.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153563.log.gz 2024-04-25T04:46:02.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44991.log 2024-04-25T04:46:02.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118950.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118950.log.gz 2024-04-25T04:46:02.547 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45670.log 2024-04-25T04:46:02.547 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44991.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44991.log.gz 2024-04-25T04:46:02.548 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139853.log 2024-04-25T04:46:02.548 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45670.log.gz 2024-04-25T04:46:02.548 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79904.log 2024-04-25T04:46:02.549 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139853.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139853.log.gz 2024-04-25T04:46:02.549 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157616.log 2024-04-25T04:46:02.550 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79904.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79904.log.gz 2024-04-25T04:46:02.550 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52545.log 2024-04-25T04:46:02.550 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157616.log.gz 2024-04-25T04:46:02.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48242.log 2024-04-25T04:46:02.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52545.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52545.log.gz 2024-04-25T04:46:02.552 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114273.log 2024-04-25T04:46:02.552 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48242.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48242.log.gz 2024-04-25T04:46:02.552 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142777.log 2024-04-25T04:46:02.553 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.114273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114273.log.gz 2024-04-25T04:46:02.553 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126208.log 2024-04-25T04:46:02.554 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142777.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142777.log.gz 2024-04-25T04:46:02.554 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108044.log 2024-04-25T04:46:02.555 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126208.log.gz 2024-04-25T04:46:02.555 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82638.log 2024-04-25T04:46:02.556 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108044.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108044.log.gz 2024-04-25T04:46:02.556 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114478.log 2024-04-25T04:46:02.556 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82638.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82638.log.gz 2024-04-25T04:46:02.557 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84707.log 2024-04-25T04:46:02.557 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114478.log.gz 2024-04-25T04:46:02.557 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154158.log 2024-04-25T04:46:02.558 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84707.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84707.log.gz 2024-04-25T04:46:02.558 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148532.log 2024-04-25T04:46:02.559 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154158.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154158.log.gz 2024-04-25T04:46:02.559 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146680.log 2024-04-25T04:46:02.560 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148532.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148532.log.gz 2024-04-25T04:46:02.560 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73164.log 2024-04-25T04:46:02.561 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146680.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146680.log.gz 2024-04-25T04:46:02.561 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115711.log 2024-04-25T04:46:02.561 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73164.log.gz 2024-04-25T04:46:02.562 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108585.log 2024-04-25T04:46:02.562 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115711.log.gz 2024-04-25T04:46:02.562 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57304.log 2024-04-25T04:46:02.563 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108585.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108585.log.gz 2024-04-25T04:46:02.563 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96403.log 2024-04-25T04:46:02.564 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57304.log.gz 2024-04-25T04:46:02.564 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155607.log 2024-04-25T04:46:02.565 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96403.log.gz 2024-04-25T04:46:02.565 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72619.log 2024-04-25T04:46:02.566 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155607.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155607.log.gz 2024-04-25T04:46:02.566 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112851.log 2024-04-25T04:46:02.566 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72619.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72619.log.gz 2024-04-25T04:46:02.567 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36472.log 2024-04-25T04:46:02.567 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112851.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112851.log.gz 2024-04-25T04:46:02.568 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51930.log 2024-04-25T04:46:02.568 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.36472.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36472.log.gz 2024-04-25T04:46:02.568 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53338.log 2024-04-25T04:46:02.569 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51930.log.gz 2024-04-25T04:46:02.569 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110849.log 2024-04-25T04:46:02.570 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53338.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53338.log.gz 2024-04-25T04:46:02.570 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40607.log 2024-04-25T04:46:02.571 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110849.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110849.log.gz 2024-04-25T04:46:02.571 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128806.log 2024-04-25T04:46:02.572 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40607.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40607.log.gz 2024-04-25T04:46:02.572 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48490.log 2024-04-25T04:46:02.572 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128806.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128806.log.gz 2024-04-25T04:46:02.573 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108257.log 2024-04-25T04:46:02.573 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48490.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48490.log.gz 2024-04-25T04:46:02.573 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66502.log 2024-04-25T04:46:02.574 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108257.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108257.log.gz 2024-04-25T04:46:02.574 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43095.log 2024-04-25T04:46:02.575 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66502.log.gz 2024-04-25T04:46:02.575 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142988.log 2024-04-25T04:46:02.576 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43095.log.gz 2024-04-25T04:46:02.576 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106480.log 2024-04-25T04:46:02.577 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142988.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142988.log.gz 2024-04-25T04:46:02.577 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119902.log 2024-04-25T04:46:02.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106480.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106480.log.gz 2024-04-25T04:46:02.578 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116882.log 2024-04-25T04:46:02.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119902.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119902.log.gz 2024-04-25T04:46:02.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115049.log 2024-04-25T04:46:02.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116882.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116882.log.gz 2024-04-25T04:46:02.580 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85792.log 2024-04-25T04:46:02.580 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115049.log.gz 2024-04-25T04:46:02.581 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61912.log 2024-04-25T04:46:02.582 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85792.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85792.log.gz 2024-04-25T04:46:02.582 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142286.log 2024-04-25T04:46:02.582 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61912.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61912.log.gz 2024-04-25T04:46:02.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115021.log 2024-04-25T04:46:02.583 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142286.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142286.log.gz 2024-04-25T04:46:02.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125562.log 2024-04-25T04:46:02.584 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115021.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115021.log.gz 2024-04-25T04:46:02.584 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153226.log 2024-04-25T04:46:02.585 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125562.log.gz 2024-04-25T04:46:02.585 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139641.log 2024-04-25T04:46:02.586 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153226.log.gz 2024-04-25T04:46:02.586 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153123.log 2024-04-25T04:46:02.587 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139641.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139641.log.gz 2024-04-25T04:46:02.587 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127958.log 2024-04-25T04:46:02.587 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153123.log.gz 2024-04-25T04:46:02.588 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58881.log 2024-04-25T04:46:02.588 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127958.log.gz 2024-04-25T04:46:02.588 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134199.log 2024-04-25T04:46:02.589 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58881.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58881.log.gz 2024-04-25T04:46:02.589 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75377.log 2024-04-25T04:46:02.590 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134199.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134199.log.gz 2024-04-25T04:46:02.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49544.log 2024-04-25T04:46:02.591 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75377.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75377.log.gz 2024-04-25T04:46:02.591 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52457.log 2024-04-25T04:46:02.592 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49544.log.gz 2024-04-25T04:46:02.592 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87802.log 2024-04-25T04:46:02.592 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52457.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52457.log.gz 2024-04-25T04:46:02.593 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136585.log 2024-04-25T04:46:02.593 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.87802.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87802.log.gz 2024-04-25T04:46:02.594 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136652.log 2024-04-25T04:46:02.594 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136585.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136585.log.gz 2024-04-25T04:46:02.595 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101841.log 2024-04-25T04:46:02.595 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136652.log.gz 2024-04-25T04:46:02.595 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67672.log 2024-04-25T04:46:02.596 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101841.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101841.log.gz 2024-04-25T04:46:02.596 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43059.log 2024-04-25T04:46:02.597 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67672.log.gz 2024-04-25T04:46:02.597 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109364.log 2024-04-25T04:46:02.597 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43059.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43059.log.gz 2024-04-25T04:46:02.598 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103261.log 2024-04-25T04:46:02.598 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109364.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109364.log.gz 2024-04-25T04:46:02.599 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91320.log 2024-04-25T04:46:02.599 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103261.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103261.log.gz 2024-04-25T04:46:02.600 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149385.log 2024-04-25T04:46:02.600 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.91320.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91320.log.gz 2024-04-25T04:46:02.600 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60064.log 2024-04-25T04:46:02.601 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149385.log.gz 2024-04-25T04:46:02.601 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140918.log 2024-04-25T04:46:02.602 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60064.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60064.log.gz 2024-04-25T04:46:02.602 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128231.log 2024-04-25T04:46:02.603 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140918.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140918.log.gz 2024-04-25T04:46:02.603 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87894.log 2024-04-25T04:46:02.604 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128231.log.gz 2024-04-25T04:46:02.604 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92690.log 2024-04-25T04:46:02.605 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87894.log.gz 2024-04-25T04:46:02.605 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157999.log 2024-04-25T04:46:02.606 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92690.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92690.log.gz 2024-04-25T04:46:02.606 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59894.log 2024-04-25T04:46:02.606 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157999.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157999.log.gz 2024-04-25T04:46:02.607 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61588.log 2024-04-25T04:46:02.607 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59894.log.gz 2024-04-25T04:46:02.608 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123820.log 2024-04-25T04:46:02.608 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61588.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61588.log.gz 2024-04-25T04:46:02.609 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108670.log 2024-04-25T04:46:02.609 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123820.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123820.log.gz 2024-04-25T04:46:02.609 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67983.log 2024-04-25T04:46:02.610 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108670.log.gz 2024-04-25T04:46:02.610 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104674.log 2024-04-25T04:46:02.611 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67983.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67983.log.gz 2024-04-25T04:46:02.611 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141992.log 2024-04-25T04:46:02.612 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104674.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104674.log.gz 2024-04-25T04:46:02.612 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70259.log 2024-04-25T04:46:02.613 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141992.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141992.log.gz 2024-04-25T04:46:02.613 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143281.log 2024-04-25T04:46:02.613 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70259.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70259.log.gz 2024-04-25T04:46:02.614 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50122.log 2024-04-25T04:46:02.614 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143281.log.gz 2024-04-25T04:46:02.615 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69998.log 2024-04-25T04:46:02.615 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50122.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50122.log.gz 2024-04-25T04:46:02.615 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64418.log 2024-04-25T04:46:02.616 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69998.log.gz 2024-04-25T04:46:02.616 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77447.log 2024-04-25T04:46:02.617 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64418.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64418.log.gz 2024-04-25T04:46:02.617 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122873.log 2024-04-25T04:46:02.618 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77447.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77447.log.gz 2024-04-25T04:46:02.618 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146956.log 2024-04-25T04:46:02.618 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122873.log.gz 2024-04-25T04:46:02.619 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158757.log 2024-04-25T04:46:02.619 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.146956.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146956.log.gz 2024-04-25T04:46:02.620 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149020.log 2024-04-25T04:46:02.620 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.158757.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158757.log.gz 2024-04-25T04:46:02.621 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91119.log 2024-04-25T04:46:02.621 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149020.log.gz 2024-04-25T04:46:02.622 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46340.log 2024-04-25T04:46:02.622 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91119.log.gz 2024-04-25T04:46:02.623 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67984.log 2024-04-25T04:46:02.623 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46340.log.gz 2024-04-25T04:46:02.623 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87680.log 2024-04-25T04:46:02.624 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67984.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67984.log.gz 2024-04-25T04:46:02.624 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117756.log 2024-04-25T04:46:02.625 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87680.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87680.log.gz 2024-04-25T04:46:02.625 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73084.log 2024-04-25T04:46:02.626 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117756.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117756.log.gz 2024-04-25T04:46:02.626 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64865.log 2024-04-25T04:46:02.627 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73084.log.gz 2024-04-25T04:46:02.627 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107624.log 2024-04-25T04:46:02.627 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64865.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64865.log.gz 2024-04-25T04:46:02.628 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89247.log 2024-04-25T04:46:02.628 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107624.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107624.log.gz 2024-04-25T04:46:02.628 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143114.log 2024-04-25T04:46:02.629 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89247.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89247.log.gz 2024-04-25T04:46:02.629 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85020.log 2024-04-25T04:46:02.630 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143114.log.gz 2024-04-25T04:46:02.630 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121083.log 2024-04-25T04:46:02.631 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85020.log.gz 2024-04-25T04:46:02.631 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128489.log 2024-04-25T04:46:02.632 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121083.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121083.log.gz 2024-04-25T04:46:02.632 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127915.log 2024-04-25T04:46:02.632 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128489.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128489.log.gz 2024-04-25T04:46:02.633 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140162.log 2024-04-25T04:46:02.633 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127915.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127915.log.gz 2024-04-25T04:46:02.633 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63227.log 2024-04-25T04:46:02.634 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140162.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140162.log.gz 2024-04-25T04:46:02.634 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60693.log 2024-04-25T04:46:02.635 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63227.log.gz 2024-04-25T04:46:02.635 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114034.log 2024-04-25T04:46:02.636 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60693.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60693.log.gz 2024-04-25T04:46:02.636 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128270.log 2024-04-25T04:46:02.637 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114034.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114034.log.gz 2024-04-25T04:46:02.637 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110983.log 2024-04-25T04:46:02.637 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128270.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128270.log.gz 2024-04-25T04:46:02.638 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69629.log 2024-04-25T04:46:02.638 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110983.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110983.log.gz 2024-04-25T04:46:02.639 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75228.log 2024-04-25T04:46:02.639 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69629.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69629.log.gz 2024-04-25T04:46:02.639 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114865.log 2024-04-25T04:46:02.640 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75228.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75228.log.gz 2024-04-25T04:46:02.640 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75185.log 2024-04-25T04:46:02.641 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114865.log.gz 2024-04-25T04:46:02.641 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69546.log 2024-04-25T04:46:02.642 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75185.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75185.log.gz 2024-04-25T04:46:02.642 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71990.log 2024-04-25T04:46:02.643 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69546.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69546.log.gz 2024-04-25T04:46:02.643 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80824.log 2024-04-25T04:46:02.643 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71990.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71990.log.gz 2024-04-25T04:46:02.644 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125234.log 2024-04-25T04:46:02.644 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80824.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80824.log.gz 2024-04-25T04:46:02.645 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43606.log 2024-04-25T04:46:02.645 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125234.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125234.log.gz 2024-04-25T04:46:02.645 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80008.log 2024-04-25T04:46:02.646 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43606.log.gz 2024-04-25T04:46:02.646 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105598.log 2024-04-25T04:46:02.647 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80008.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80008.log.gz 2024-04-25T04:46:02.647 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55035.log 2024-04-25T04:46:02.648 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105598.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105598.log.gz 2024-04-25T04:46:02.648 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133667.log 2024-04-25T04:46:02.649 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55035.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55035.log.gz 2024-04-25T04:46:02.649 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160250.log 2024-04-25T04:46:02.649 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133667.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133667.log.gz 2024-04-25T04:46:02.650 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74114.log 2024-04-25T04:46:02.650 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160250.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160250.log.gz 2024-04-25T04:46:02.651 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111514.log 2024-04-25T04:46:02.651 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.74114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74114.log.gz 2024-04-25T04:46:02.652 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73371.log 2024-04-25T04:46:02.652 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111514.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111514.log.gz 2024-04-25T04:46:02.652 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113543.log 2024-04-25T04:46:02.653 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73371.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73371.log.gz 2024-04-25T04:46:02.653 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67060.log 2024-04-25T04:46:02.654 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113543.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113543.log.gz 2024-04-25T04:46:02.654 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71823.log 2024-04-25T04:46:02.655 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67060.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67060.log.gz 2024-04-25T04:46:02.655 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129645.log 2024-04-25T04:46:02.655 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71823.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71823.log.gz 2024-04-25T04:46:02.656 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159575.log 2024-04-25T04:46:02.656 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129645.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129645.log.gz 2024-04-25T04:46:02.657 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68594.log 2024-04-25T04:46:02.657 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159575.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159575.log.gz 2024-04-25T04:46:02.658 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48601.log 2024-04-25T04:46:02.658 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68594.log.gz 2024-04-25T04:46:02.659 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142987.log 2024-04-25T04:46:02.659 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48601.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48601.log.gz 2024-04-25T04:46:02.660 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59382.log 2024-04-25T04:46:02.660 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142987.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142987.log.gz 2024-04-25T04:46:02.660 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43896.log 2024-04-25T04:46:02.661 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59382.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59382.log.gz 2024-04-25T04:46:02.661 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79692.log 2024-04-25T04:46:02.662 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43896.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43896.log.gz 2024-04-25T04:46:02.662 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55151.log 2024-04-25T04:46:02.663 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79692.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79692.log.gz 2024-04-25T04:46:02.663 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91706.log 2024-04-25T04:46:02.664 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55151.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55151.log.gz 2024-04-25T04:46:02.664 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128813.log 2024-04-25T04:46:02.664 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91706.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91706.log.gz 2024-04-25T04:46:02.665 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116874.log 2024-04-25T04:46:02.665 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128813.log.gz 2024-04-25T04:46:02.666 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149625.log 2024-04-25T04:46:02.666 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116874.log.gz 2024-04-25T04:46:02.667 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44505.log 2024-04-25T04:46:02.667 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149625.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149625.log.gz 2024-04-25T04:46:02.667 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113523.log 2024-04-25T04:46:02.668 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44505.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44505.log.gz 2024-04-25T04:46:02.668 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158925.log 2024-04-25T04:46:02.669 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113523.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113523.log.gz 2024-04-25T04:46:02.669 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61199.log 2024-04-25T04:46:02.670 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158925.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158925.log.gz 2024-04-25T04:46:02.670 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83203.log 2024-04-25T04:46:02.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61199.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61199.log.gz 2024-04-25T04:46:02.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129248.log 2024-04-25T04:46:02.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83203.log.gz 2024-04-25T04:46:02.672 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91453.log 2024-04-25T04:46:02.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129248.log.gz 2024-04-25T04:46:02.673 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108610.log 2024-04-25T04:46:02.674 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91453.log.gz 2024-04-25T04:46:02.674 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127791.log 2024-04-25T04:46:02.674 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108610.log.gz 2024-04-25T04:46:02.674 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61891.log 2024-04-25T04:46:02.675 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127791.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127791.log.gz 2024-04-25T04:46:02.675 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52099.log 2024-04-25T04:46:02.676 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61891.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61891.log.gz 2024-04-25T04:46:02.676 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105342.log 2024-04-25T04:46:02.677 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52099.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52099.log.gz 2024-04-25T04:46:02.677 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87789.log 2024-04-25T04:46:02.678 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105342.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105342.log.gz 2024-04-25T04:46:02.678 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84980.log 2024-04-25T04:46:02.679 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.87789.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87789.log.gz 2024-04-25T04:46:02.679 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92804.log 2024-04-25T04:46:02.679 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84980.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84980.log.gz 2024-04-25T04:46:02.680 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71697.log 2024-04-25T04:46:02.680 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92804.log.gz 2024-04-25T04:46:02.681 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68407.log 2024-04-25T04:46:02.681 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71697.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71697.log.gz 2024-04-25T04:46:02.681 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74112.log 2024-04-25T04:46:02.682 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68407.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68407.log.gz 2024-04-25T04:46:02.682 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108855.log 2024-04-25T04:46:02.683 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.74112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74112.log.gz 2024-04-25T04:46:02.683 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61281.log 2024-04-25T04:46:02.684 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108855.log.gz 2024-04-25T04:46:02.684 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51238.log 2024-04-25T04:46:02.685 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61281.log.gz 2024-04-25T04:46:02.685 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128168.log 2024-04-25T04:46:02.686 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51238.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51238.log.gz 2024-04-25T04:46:02.686 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115898.log 2024-04-25T04:46:02.686 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128168.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128168.log.gz 2024-04-25T04:46:02.687 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106481.log 2024-04-25T04:46:02.688 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115898.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115898.log.gz 2024-04-25T04:46:02.688 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79628.log 2024-04-25T04:46:02.688 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106481.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106481.log.gz 2024-04-25T04:46:02.689 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47350.log 2024-04-25T04:46:02.689 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79628.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79628.log.gz 2024-04-25T04:46:02.690 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102790.log 2024-04-25T04:46:02.691 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47350.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47350.log.gz 2024-04-25T04:46:02.691 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101047.log 2024-04-25T04:46:02.691 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102790.log.gz 2024-04-25T04:46:02.691 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137127.log 2024-04-25T04:46:02.692 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101047.log.gz 2024-04-25T04:46:02.692 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79118.log 2024-04-25T04:46:02.693 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137127.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137127.log.gz 2024-04-25T04:46:02.693 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73043.log 2024-04-25T04:46:02.694 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.79118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79118.log.gz 2024-04-25T04:46:02.694 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150660.log 2024-04-25T04:46:02.695 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73043.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73043.log.gz 2024-04-25T04:46:02.695 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64703.log 2024-04-25T04:46:02.696 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150660.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150660.log.gz 2024-04-25T04:46:02.696 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46278.log 2024-04-25T04:46:02.697 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.64703.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64703.log.gz 2024-04-25T04:46:02.697 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108947.log 2024-04-25T04:46:02.697 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46278.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46278.log.gz 2024-04-25T04:46:02.698 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113274.log 2024-04-25T04:46:02.698 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108947.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108947.log.gz 2024-04-25T04:46:02.698 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130380.log 2024-04-25T04:46:02.699 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113274.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113274.log.gz 2024-04-25T04:46:02.699 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99980.log 2024-04-25T04:46:02.700 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130380.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130380.log.gz 2024-04-25T04:46:02.700 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111252.log 2024-04-25T04:46:02.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99980.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99980.log.gz 2024-04-25T04:46:02.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74956.log 2024-04-25T04:46:02.702 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111252.log.gz 2024-04-25T04:46:02.702 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72410.log 2024-04-25T04:46:02.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74956.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74956.log.gz 2024-04-25T04:46:02.703 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75088.log 2024-04-25T04:46:02.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72410.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72410.log.gz 2024-04-25T04:46:02.704 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100905.log 2024-04-25T04:46:02.704 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75088.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75088.log.gz 2024-04-25T04:46:02.705 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99477.log 2024-04-25T04:46:02.705 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.100905.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100905.log.gz 2024-04-25T04:46:02.706 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139744.log 2024-04-25T04:46:02.706 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99477.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99477.log.gz 2024-04-25T04:46:02.706 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124098.log 2024-04-25T04:46:02.707 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139744.log.gz 2024-04-25T04:46:02.707 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79587.log 2024-04-25T04:46:02.708 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124098.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124098.log.gz 2024-04-25T04:46:02.708 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94521.log 2024-04-25T04:46:02.709 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79587.log.gz 2024-04-25T04:46:02.709 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95333.log 2024-04-25T04:46:02.710 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94521.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94521.log.gz 2024-04-25T04:46:02.710 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64263.log 2024-04-25T04:46:02.711 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95333.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95333.log.gz 2024-04-25T04:46:02.711 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70615.log 2024-04-25T04:46:02.712 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64263.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64263.log.gz 2024-04-25T04:46:02.712 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115230.log 2024-04-25T04:46:02.713 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70615.log.gz 2024-04-25T04:46:02.713 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130630.log 2024-04-25T04:46:02.714 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115230.log.gz 2024-04-25T04:46:02.714 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111160.log 2024-04-25T04:46:02.715 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130630.log.gz 2024-04-25T04:46:02.715 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77809.log 2024-04-25T04:46:02.715 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.111160.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111160.log.gz 2024-04-25T04:46:02.716 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56204.log 2024-04-25T04:46:02.716 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77809.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77809.log.gz 2024-04-25T04:46:02.717 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59258.log 2024-04-25T04:46:02.717 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56204.log.gz 2024-04-25T04:46:02.717 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113201.log 2024-04-25T04:46:02.718 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59258.log.gz 2024-04-25T04:46:02.718 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149909.log 2024-04-25T04:46:02.719 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113201.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113201.log.gz 2024-04-25T04:46:02.719 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67925.log 2024-04-25T04:46:02.720 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149909.log.gz 2024-04-25T04:46:02.720 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151829.log 2024-04-25T04:46:02.721 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67925.log.gz 2024-04-25T04:46:02.721 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151994.log 2024-04-25T04:46:02.721 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151829.log.gz 2024-04-25T04:46:02.722 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65367.log 2024-04-25T04:46:02.722 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151994.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151994.log.gz 2024-04-25T04:46:02.723 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141172.log 2024-04-25T04:46:02.723 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65367.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65367.log.gz 2024-04-25T04:46:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123424.log 2024-04-25T04:46:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141172.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141172.log.gz 2024-04-25T04:46:02.724 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115919.log 2024-04-25T04:46:02.725 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123424.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123424.log.gz 2024-04-25T04:46:02.725 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49384.log 2024-04-25T04:46:02.726 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115919.log.gz 2024-04-25T04:46:02.726 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.122981.log 2024-04-25T04:46:02.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49384.log.gz 2024-04-25T04:46:02.727 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87602.log 2024-04-25T04:46:02.728 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.122981.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.122981.log.gz 2024-04-25T04:46:02.728 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70344.log 2024-04-25T04:46:02.728 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87602.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87602.log.gz 2024-04-25T04:46:02.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119986.log 2024-04-25T04:46:02.730 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70344.log.gz 2024-04-25T04:46:02.730 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110264.log 2024-04-25T04:46:02.731 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119986.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119986.log.gz 2024-04-25T04:46:02.731 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127873.log 2024-04-25T04:46:02.731 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110264.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110264.log.gz 2024-04-25T04:46:02.732 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94880.log 2024-04-25T04:46:02.732 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127873.log.gz 2024-04-25T04:46:02.733 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153873.log 2024-04-25T04:46:02.733 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94880.log.gz 2024-04-25T04:46:02.733 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42403.log 2024-04-25T04:46:02.734 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153873.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153873.log.gz 2024-04-25T04:46:02.734 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97051.log 2024-04-25T04:46:02.735 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42403.log.gz 2024-04-25T04:46:02.735 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151853.log 2024-04-25T04:46:02.736 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97051.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97051.log.gz 2024-04-25T04:46:02.736 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55680.log 2024-04-25T04:46:02.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151853.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151853.log.gz 2024-04-25T04:46:02.737 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115985.log 2024-04-25T04:46:02.738 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55680.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55680.log.gz 2024-04-25T04:46:02.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150154.log 2024-04-25T04:46:02.739 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115985.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115985.log.gz 2024-04-25T04:46:02.739 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88138.log 2024-04-25T04:46:02.739 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.150154.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150154.log.gz 2024-04-25T04:46:02.740 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150508.log 2024-04-25T04:46:02.740 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88138.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88138.log.gz 2024-04-25T04:46:02.741 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62349.log 2024-04-25T04:46:02.741 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150508.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150508.log.gz 2024-04-25T04:46:02.741 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88329.log 2024-04-25T04:46:02.742 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62349.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62349.log.gz 2024-04-25T04:46:02.742 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114563.log 2024-04-25T04:46:02.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88329.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88329.log.gz 2024-04-25T04:46:02.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67342.log 2024-04-25T04:46:02.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114563.log.gz 2024-04-25T04:46:02.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151791.log 2024-04-25T04:46:02.745 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67342.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67342.log.gz 2024-04-25T04:46:02.745 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77768.log 2024-04-25T04:46:02.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151791.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151791.log.gz 2024-04-25T04:46:02.746 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52001.log 2024-04-25T04:46:02.747 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77768.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77768.log.gz 2024-04-25T04:46:02.747 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156203.log 2024-04-25T04:46:02.747 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52001.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52001.log.gz 2024-04-25T04:46:02.748 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66572.log 2024-04-25T04:46:02.748 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156203.log.gz 2024-04-25T04:46:02.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144771.log 2024-04-25T04:46:02.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66572.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66572.log.gz 2024-04-25T04:46:02.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120872.log 2024-04-25T04:46:02.750 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144771.log.gz 2024-04-25T04:46:02.750 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45267.log 2024-04-25T04:46:02.751 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120872.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120872.log.gz 2024-04-25T04:46:02.751 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107063.log 2024-04-25T04:46:02.752 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45267.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45267.log.gz 2024-04-25T04:46:02.752 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142348.log 2024-04-25T04:46:02.753 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107063.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107063.log.gz 2024-04-25T04:46:02.753 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94209.log 2024-04-25T04:46:02.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142348.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142348.log.gz 2024-04-25T04:46:02.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130181.log 2024-04-25T04:46:02.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94209.log.gz 2024-04-25T04:46:02.755 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160082.log 2024-04-25T04:46:02.755 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130181.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130181.log.gz 2024-04-25T04:46:02.756 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144373.log 2024-04-25T04:46:02.756 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160082.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160082.log.gz 2024-04-25T04:46:02.757 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155838.log 2024-04-25T04:46:02.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144373.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144373.log.gz 2024-04-25T04:46:02.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54926.log 2024-04-25T04:46:02.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155838.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155838.log.gz 2024-04-25T04:46:02.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141340.log 2024-04-25T04:46:02.759 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54926.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54926.log.gz 2024-04-25T04:46:02.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82556.log 2024-04-25T04:46:02.760 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141340.log.gz 2024-04-25T04:46:02.760 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103785.log 2024-04-25T04:46:02.761 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82556.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82556.log.gz 2024-04-25T04:46:02.761 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153733.log 2024-04-25T04:46:02.762 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103785.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103785.log.gz 2024-04-25T04:46:02.762 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52591.log 2024-04-25T04:46:02.763 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153733.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153733.log.gz 2024-04-25T04:46:02.763 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160055.log 2024-04-25T04:46:02.763 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52591.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52591.log.gz 2024-04-25T04:46:02.764 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82868.log 2024-04-25T04:46:02.765 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160055.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160055.log.gz 2024-04-25T04:46:02.765 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79415.log 2024-04-25T04:46:02.765 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82868.log.gz 2024-04-25T04:46:02.766 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100235.log 2024-04-25T04:46:02.766 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79415.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79415.log.gz 2024-04-25T04:46:02.766 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143537.log 2024-04-25T04:46:02.767 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100235.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100235.log.gz 2024-04-25T04:46:02.767 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118825.log 2024-04-25T04:46:02.768 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143537.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143537.log.gz 2024-04-25T04:46:02.768 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46276.log 2024-04-25T04:46:02.769 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118825.log.gz 2024-04-25T04:46:02.769 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68467.log 2024-04-25T04:46:02.770 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46276.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46276.log.gz 2024-04-25T04:46:02.770 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80823.log 2024-04-25T04:46:02.771 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68467.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68467.log.gz 2024-04-25T04:46:02.771 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119492.log 2024-04-25T04:46:02.771 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80823.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80823.log.gz 2024-04-25T04:46:02.772 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107246.log 2024-04-25T04:46:02.772 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119492.log.gz 2024-04-25T04:46:02.773 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95373.log 2024-04-25T04:46:02.773 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107246.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107246.log.gz 2024-04-25T04:46:02.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86600.log 2024-04-25T04:46:02.774 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95373.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95373.log.gz 2024-04-25T04:46:02.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36232.log 2024-04-25T04:46:02.775 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86600.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86600.log.gz 2024-04-25T04:46:02.775 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72451.log 2024-04-25T04:46:02.776 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.36232.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36232.log.gz 2024-04-25T04:46:02.776 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94933.log 2024-04-25T04:46:02.777 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72451.log.gz 2024-04-25T04:46:02.777 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132663.log 2024-04-25T04:46:02.778 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94933.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94933.log.gz 2024-04-25T04:46:02.778 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72618.log 2024-04-25T04:46:02.779 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132663.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132663.log.gz 2024-04-25T04:46:02.779 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135918.log 2024-04-25T04:46:02.779 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72618.log.gz 2024-04-25T04:46:02.779 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130423.log 2024-04-25T04:46:02.780 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135918.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135918.log.gz 2024-04-25T04:46:02.780 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40606.log 2024-04-25T04:46:02.781 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130423.log.gz 2024-04-25T04:46:02.781 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115332.log 2024-04-25T04:46:02.782 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40606.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40606.log.gz 2024-04-25T04:46:02.782 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116725.log 2024-04-25T04:46:02.783 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115332.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115332.log.gz 2024-04-25T04:46:02.783 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149871.log 2024-04-25T04:46:02.784 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116725.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116725.log.gz 2024-04-25T04:46:02.784 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84413.log 2024-04-25T04:46:02.785 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149871.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149871.log.gz 2024-04-25T04:46:02.785 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122376.log 2024-04-25T04:46:02.785 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84413.log.gz 2024-04-25T04:46:02.786 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117060.log 2024-04-25T04:46:02.786 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122376.log.gz 2024-04-25T04:46:02.787 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161411.log 2024-04-25T04:46:02.787 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117060.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117060.log.gz 2024-04-25T04:46:02.788 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133382.log 2024-04-25T04:46:02.788 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161411.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161411.log.gz 2024-04-25T04:46:02.788 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139226.log 2024-04-25T04:46:02.789 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133382.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133382.log.gz 2024-04-25T04:46:02.789 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49935.log 2024-04-25T04:46:02.790 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139226.log.gz 2024-04-25T04:46:02.790 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97599.log 2024-04-25T04:46:02.791 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.49935.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49935.log.gz 2024-04-25T04:46:02.791 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69375.log 2024-04-25T04:46:02.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97599.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97599.log.gz 2024-04-25T04:46:02.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90878.log 2024-04-25T04:46:02.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69375.log.gz 2024-04-25T04:46:02.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48240.log 2024-04-25T04:46:02.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90878.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90878.log.gz 2024-04-25T04:46:02.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79343.log 2024-04-25T04:46:02.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48240.log.gz 2024-04-25T04:46:02.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90786.log 2024-04-25T04:46:02.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79343.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79343.log.gz 2024-04-25T04:46:02.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136953.log 2024-04-25T04:46:02.796 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90786.log.gz 2024-04-25T04:46:02.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79232.log 2024-04-25T04:46:02.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136953.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136953.log.gz 2024-04-25T04:46:02.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136463.log 2024-04-25T04:46:02.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79232.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79232.log.gz 2024-04-25T04:46:02.798 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133217.log 2024-04-25T04:46:02.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136463.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136463.log.gz 2024-04-25T04:46:02.799 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80571.log 2024-04-25T04:46:02.799 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133217.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133217.log.gz 2024-04-25T04:46:02.799 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110331.log 2024-04-25T04:46:02.800 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80571.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80571.log.gz 2024-04-25T04:46:02.800 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80177.log 2024-04-25T04:46:02.801 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110331.log.gz 2024-04-25T04:46:02.801 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157254.log 2024-04-25T04:46:02.802 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80177.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80177.log.gz 2024-04-25T04:46:02.802 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121436.log 2024-04-25T04:46:02.803 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157254.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157254.log.gz 2024-04-25T04:46:02.803 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68929.log 2024-04-25T04:46:02.803 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121436.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121436.log.gz 2024-04-25T04:46:02.804 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122114.log 2024-04-25T04:46:02.804 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68929.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68929.log.gz 2024-04-25T04:46:02.805 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91454.log 2024-04-25T04:46:02.805 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122114.log.gz 2024-04-25T04:46:02.805 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74184.log 2024-04-25T04:46:02.806 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91454.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91454.log.gz 2024-04-25T04:46:02.806 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130588.log 2024-04-25T04:46:02.807 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74184.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74184.log.gz 2024-04-25T04:46:02.807 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152678.log 2024-04-25T04:46:02.808 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130588.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130588.log.gz 2024-04-25T04:46:02.808 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117123.log 2024-04-25T04:46:02.809 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152678.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152678.log.gz 2024-04-25T04:46:02.809 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47571.log 2024-04-25T04:46:02.809 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117123.log.gz 2024-04-25T04:46:02.810 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152388.log 2024-04-25T04:46:02.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47571.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47571.log.gz 2024-04-25T04:46:02.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126608.log 2024-04-25T04:46:02.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152388.log.gz 2024-04-25T04:46:02.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96491.log 2024-04-25T04:46:02.812 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.126608.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126608.log.gz 2024-04-25T04:46:02.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46255.log 2024-04-25T04:46:02.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96491.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96491.log.gz 2024-04-25T04:46:02.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158438.log 2024-04-25T04:46:02.814 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46255.log.gz 2024-04-25T04:46:02.814 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96173.log 2024-04-25T04:46:02.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158438.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158438.log.gz 2024-04-25T04:46:02.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97157.log 2024-04-25T04:46:02.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96173.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96173.log.gz 2024-04-25T04:46:02.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141235.log 2024-04-25T04:46:02.816 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97157.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97157.log.gz 2024-04-25T04:46:02.817 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126410.log 2024-04-25T04:46:02.817 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141235.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141235.log.gz 2024-04-25T04:46:02.817 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161019.log 2024-04-25T04:46:02.818 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126410.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126410.log.gz 2024-04-25T04:46:02.818 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63605.log 2024-04-25T04:46:02.819 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161019.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161019.log.gz 2024-04-25T04:46:02.819 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44861.log 2024-04-25T04:46:02.820 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63605.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63605.log.gz 2024-04-25T04:46:02.820 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44673.log 2024-04-25T04:46:02.821 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44861.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44861.log.gz 2024-04-25T04:46:02.821 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49545.log 2024-04-25T04:46:02.821 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44673.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44673.log.gz 2024-04-25T04:46:02.822 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73926.log 2024-04-25T04:46:02.822 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49545.log.gz 2024-04-25T04:46:02.823 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155943.log 2024-04-25T04:46:02.823 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73926.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73926.log.gz 2024-04-25T04:46:02.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110370.log 2024-04-25T04:46:02.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155943.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155943.log.gz 2024-04-25T04:46:02.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98675.log 2024-04-25T04:46:02.825 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110370.log.gz 2024-04-25T04:46:02.825 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66039.log 2024-04-25T04:46:02.826 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98675.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98675.log.gz 2024-04-25T04:46:02.826 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101275.log 2024-04-25T04:46:02.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.66039.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66039.log.gz 2024-04-25T04:46:02.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161166.log 2024-04-25T04:46:02.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101275.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101275.log.gz 2024-04-25T04:46:02.828 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124044.log 2024-04-25T04:46:02.828 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161166.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161166.log.gz 2024-04-25T04:46:02.829 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106941.log 2024-04-25T04:46:02.829 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124044.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124044.log.gz 2024-04-25T04:46:02.830 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46527.log 2024-04-25T04:46:02.830 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106941.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106941.log.gz 2024-04-25T04:46:02.830 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71057.log 2024-04-25T04:46:02.831 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46527.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46527.log.gz 2024-04-25T04:46:02.831 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154294.log 2024-04-25T04:46:02.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71057.log.gz 2024-04-25T04:46:02.832 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56079.log 2024-04-25T04:46:02.833 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154294.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154294.log.gz 2024-04-25T04:46:02.833 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117292.log 2024-04-25T04:46:02.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56079.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56079.log.gz 2024-04-25T04:46:02.834 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144829.log 2024-04-25T04:46:02.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117292.log.gz 2024-04-25T04:46:02.835 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74155.log 2024-04-25T04:46:02.835 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144829.log.gz 2024-04-25T04:46:02.836 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131441.log 2024-04-25T04:46:02.836 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74155.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74155.log.gz 2024-04-25T04:46:02.837 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78547.log 2024-04-25T04:46:02.837 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131441.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131441.log.gz 2024-04-25T04:46:02.837 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88870.log 2024-04-25T04:46:02.838 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78547.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78547.log.gz 2024-04-25T04:46:02.838 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55258.log 2024-04-25T04:46:02.839 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.88870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88870.log.gz 2024-04-25T04:46:02.839 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47547.log 2024-04-25T04:46:02.841 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.55258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55258.log.gz 2024-04-25T04:46:02.841 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67635.log 2024-04-25T04:46:02.841 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47547.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47547.log.gz 2024-04-25T04:46:02.841 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56289.log 2024-04-25T04:46:02.842 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67635.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67635.log.gz 2024-04-25T04:46:02.842 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77190.log 2024-04-25T04:46:02.843 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56289.log.gz 2024-04-25T04:46:02.843 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96425.log 2024-04-25T04:46:02.843 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77190.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77190.log.gz 2024-04-25T04:46:02.844 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41926.log 2024-04-25T04:46:02.844 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96425.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96425.log.gz 2024-04-25T04:46:02.845 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132126.log 2024-04-25T04:46:02.846 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41926.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41926.log.gz 2024-04-25T04:46:02.846 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96989.log 2024-04-25T04:46:02.846 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132126.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132126.log.gz 2024-04-25T04:46:02.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56939.log 2024-04-25T04:46:02.847 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96989.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96989.log.gz 2024-04-25T04:46:02.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119802.log 2024-04-25T04:46:02.848 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56939.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56939.log.gz 2024-04-25T04:46:02.848 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46846.log 2024-04-25T04:46:02.849 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119802.log.gz 2024-04-25T04:46:02.849 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113482.log 2024-04-25T04:46:02.850 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46846.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46846.log.gz 2024-04-25T04:46:02.850 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78585.log 2024-04-25T04:46:02.851 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113482.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113482.log.gz 2024-04-25T04:46:02.851 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131502.log 2024-04-25T04:46:02.851 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78585.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78585.log.gz 2024-04-25T04:46:02.852 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93810.log 2024-04-25T04:46:02.854 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131502.log.gz 2024-04-25T04:46:02.854 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83438.log 2024-04-25T04:46:02.854 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93810.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93810.log.gz 2024-04-25T04:46:02.854 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129753.log 2024-04-25T04:46:02.854 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83438.log.gz 2024-04-25T04:46:02.855 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54312.log 2024-04-25T04:46:02.855 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129753.log.gz 2024-04-25T04:46:02.855 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116232.log 2024-04-25T04:46:02.856 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54312.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54312.log.gz 2024-04-25T04:46:02.856 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73543.log 2024-04-25T04:46:02.857 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116232.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116232.log.gz 2024-04-25T04:46:02.857 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147446.log 2024-04-25T04:46:02.858 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73543.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73543.log.gz 2024-04-25T04:46:02.858 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119925.log 2024-04-25T04:46:02.858 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147446.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147446.log.gz 2024-04-25T04:46:02.859 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119150.log 2024-04-25T04:46:02.859 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119925.log.gz 2024-04-25T04:46:02.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135442.log 2024-04-25T04:46:02.860 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119150.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119150.log.gz 2024-04-25T04:46:02.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149542.log 2024-04-25T04:46:02.861 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.135442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141446.log 2024-04-25T04:46:02.861 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135442.log.gz 2024-04-25T04:46:02.862 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149542.log.gz 2024-04-25T04:46:02.862 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45546.log 2024-04-25T04:46:02.863 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141446.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141446.log.gz 2024-04-25T04:46:02.863 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151287.log 2024-04-25T04:46:02.864 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45546.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45546.log.gz 2024-04-25T04:46:02.864 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152278.log 2024-04-25T04:46:02.864 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151287.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151287.log.gz 2024-04-25T04:46:02.865 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74953.log 2024-04-25T04:46:02.866 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152278.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152278.log.gz 2024-04-25T04:46:02.866 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84708.log 2024-04-25T04:46:02.867 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74953.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74953.log.gz 2024-04-25T04:46:02.867 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83672.log 2024-04-25T04:46:02.867 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84708.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84708.log.gz 2024-04-25T04:46:02.867 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114359.log 2024-04-25T04:46:02.868 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83672.log.gz 2024-04-25T04:46:02.868 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61303.log 2024-04-25T04:46:02.869 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114359.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114359.log.gz 2024-04-25T04:46:02.869 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139224.log 2024-04-25T04:46:02.870 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61303.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61303.log.gz 2024-04-25T04:46:02.870 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146878.log 2024-04-25T04:46:02.871 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139224.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139224.log.gz 2024-04-25T04:46:02.871 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82764.log 2024-04-25T04:46:02.871 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146878.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146878.log.gz 2024-04-25T04:46:02.872 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96276.log 2024-04-25T04:46:02.872 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82764.log.gz 2024-04-25T04:46:02.873 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59299.log 2024-04-25T04:46:02.873 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96276.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96276.log.gz 2024-04-25T04:46:02.873 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87254.log 2024-04-25T04:46:02.874 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59299.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59299.log.gz 2024-04-25T04:46:02.874 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72368.log 2024-04-25T04:46:02.875 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87254.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87254.log.gz 2024-04-25T04:46:02.875 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125748.log 2024-04-25T04:46:02.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72368.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72368.log.gz 2024-04-25T04:46:02.876 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74513.log 2024-04-25T04:46:02.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125748.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125748.log.gz 2024-04-25T04:46:02.877 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45710.log 2024-04-25T04:46:02.877 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74513.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74513.log.gz 2024-04-25T04:46:02.878 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86349.log 2024-04-25T04:46:02.878 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45710.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45710.log.gz 2024-04-25T04:46:02.878 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91097.log 2024-04-25T04:46:02.879 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86349.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86349.log.gz 2024-04-25T04:46:02.879 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69792.log 2024-04-25T04:46:02.880 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91097.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91097.log.gz 2024-04-25T04:46:02.880 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54353.log 2024-04-25T04:46:02.881 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69792.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69792.log.gz 2024-04-25T04:46:02.881 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93013.log 2024-04-25T04:46:02.882 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54353.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54353.log.gz 2024-04-25T04:46:02.882 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62161.log 2024-04-25T04:46:02.883 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93013.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93013.log.gz 2024-04-25T04:46:02.883 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65125.log 2024-04-25T04:46:02.883 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62161.log.gz 2024-04-25T04:46:02.884 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112878.log 2024-04-25T04:46:02.884 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65125.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65125.log.gz 2024-04-25T04:46:02.885 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63270.log 2024-04-25T04:46:02.885 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112878.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112878.log.gz 2024-04-25T04:46:02.886 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106041.log 2024-04-25T04:46:02.886 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63270.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63270.log.gz 2024-04-25T04:46:02.887 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96781.log 2024-04-25T04:46:02.887 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106041.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106041.log.gz 2024-04-25T04:46:02.888 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80259.log 2024-04-25T04:46:02.888 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96781.log.gz 2024-04-25T04:46:02.888 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115437.log 2024-04-25T04:46:02.889 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80259.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80259.log.gz 2024-04-25T04:46:02.889 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52269.log 2024-04-25T04:46:02.890 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115437.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115437.log.gz 2024-04-25T04:46:02.890 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123990.log 2024-04-25T04:46:02.891 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52269.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52269.log.gz 2024-04-25T04:46:02.891 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57745.log 2024-04-25T04:46:02.892 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123990.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123990.log.gz 2024-04-25T04:46:02.892 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92481.log 2024-04-25T04:46:02.893 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57745.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57745.log.gz 2024-04-25T04:46:02.893 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51475.log 2024-04-25T04:46:02.893 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92481.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92481.log.gz 2024-04-25T04:46:02.894 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161556.log 2024-04-25T04:46:02.894 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51475.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51475.log.gz 2024-04-25T04:46:02.895 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84537.log 2024-04-25T04:46:02.895 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161556.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161556.log.gz 2024-04-25T04:46:02.895 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144185.log 2024-04-25T04:46:02.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84537.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84537.log.gz 2024-04-25T04:46:02.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41960.log 2024-04-25T04:46:02.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144185.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144185.log.gz 2024-04-25T04:46:02.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67256.log 2024-04-25T04:46:02.898 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.41960.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41960.log.gz 2024-04-25T04:46:02.898 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144188.log 2024-04-25T04:46:02.899 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.67256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67256.log.gz 2024-04-25T04:46:02.899 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133085.log 2024-04-25T04:46:02.899 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144188.log.gz 2024-04-25T04:46:02.900 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60262.log 2024-04-25T04:46:02.900 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133085.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133085.log.gz 2024-04-25T04:46:02.901 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73208.log 2024-04-25T04:46:02.901 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60262.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60262.log.gz 2024-04-25T04:46:02.901 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83907.log 2024-04-25T04:46:02.902 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73208.log.gz 2024-04-25T04:46:02.902 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124811.log 2024-04-25T04:46:02.903 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83907.log.gz 2024-04-25T04:46:02.903 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146456.log 2024-04-25T04:46:02.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124811.log.gz 2024-04-25T04:46:02.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151347.log 2024-04-25T04:46:02.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146456.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146456.log.gz 2024-04-25T04:46:02.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47140.log 2024-04-25T04:46:02.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151347.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151347.log.gz 2024-04-25T04:46:02.906 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71991.log 2024-04-25T04:46:02.906 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47140.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47140.log.gz 2024-04-25T04:46:02.907 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92868.log 2024-04-25T04:46:02.907 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71991.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71991.log.gz 2024-04-25T04:46:02.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83205.log 2024-04-25T04:46:02.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92868.log.gz 2024-04-25T04:46:02.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90916.log 2024-04-25T04:46:02.909 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83205.log.gz 2024-04-25T04:46:02.909 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128271.log 2024-04-25T04:46:02.910 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90916.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90916.log.gz 2024-04-25T04:46:02.910 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99941.log 2024-04-25T04:46:02.911 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128271.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128271.log.gz 2024-04-25T04:46:02.911 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83779.log 2024-04-25T04:46:02.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99941.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99941.log.gz 2024-04-25T04:46:02.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129599.log 2024-04-25T04:46:02.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83779.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83779.log.gz 2024-04-25T04:46:02.913 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111929.log 2024-04-25T04:46:02.913 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129599.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129599.log.gz 2024-04-25T04:46:02.914 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82735.log 2024-04-25T04:46:02.914 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111929.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111929.log.gz 2024-04-25T04:46:02.914 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105137.log 2024-04-25T04:46:02.915 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82735.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82735.log.gz 2024-04-25T04:46:02.915 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153622.log 2024-04-25T04:46:02.916 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105137.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105137.log.gz 2024-04-25T04:46:02.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42198.log 2024-04-25T04:46:02.917 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153622.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153622.log.gz 2024-04-25T04:46:02.917 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104412.log 2024-04-25T04:46:02.918 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42198.log.gz 2024-04-25T04:46:02.918 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82763.log 2024-04-25T04:46:02.919 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104412.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104412.log.gz 2024-04-25T04:46:02.919 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134816.log 2024-04-25T04:46:02.919 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82763.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82763.log.gz 2024-04-25T04:46:02.920 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92669.log 2024-04-25T04:46:02.920 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134816.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134816.log.gz 2024-04-25T04:46:02.921 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59404.log 2024-04-25T04:46:02.921 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92669.log.gz 2024-04-25T04:46:02.921 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148471.log 2024-04-25T04:46:02.922 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59404.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59404.log.gz 2024-04-25T04:46:02.922 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102011.log 2024-04-25T04:46:02.923 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148471.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148471.log.gz 2024-04-25T04:46:02.924 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48533.log 2024-04-25T04:46:02.924 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102011.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102011.log.gz 2024-04-25T04:46:02.924 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131126.log 2024-04-25T04:46:02.925 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48533.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48533.log.gz 2024-04-25T04:46:02.925 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96360.log 2024-04-25T04:46:02.926 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131126.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131126.log.gz 2024-04-25T04:46:02.926 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.122785.log 2024-04-25T04:46:02.927 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96360.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96360.log.gz 2024-04-25T04:46:02.927 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47139.log 2024-04-25T04:46:02.927 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.122785.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.122785.log.gz 2024-04-25T04:46:02.928 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147695.log 2024-04-25T04:46:02.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47139.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47139.log.gz 2024-04-25T04:46:02.928 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144463.log 2024-04-25T04:46:02.929 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147695.log.gz 2024-04-25T04:46:02.929 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151923.log 2024-04-25T04:46:02.930 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144463.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144463.log.gz 2024-04-25T04:46:02.930 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64419.log 2024-04-25T04:46:02.931 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151923.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151923.log.gz 2024-04-25T04:46:02.931 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125613.log 2024-04-25T04:46:02.932 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64419.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64419.log.gz 2024-04-25T04:46:02.932 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57665.log 2024-04-25T04:46:02.932 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125613.log.gz 2024-04-25T04:46:02.933 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97288.log 2024-04-25T04:46:02.933 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57665.log.gz 2024-04-25T04:46:02.934 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99788.log 2024-04-25T04:46:02.934 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97288.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97288.log.gz 2024-04-25T04:46:02.934 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157835.log 2024-04-25T04:46:02.935 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99788.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99788.log.gz 2024-04-25T04:46:02.935 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108256.log 2024-04-25T04:46:02.936 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157835.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157835.log.gz 2024-04-25T04:46:02.936 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97499.log 2024-04-25T04:46:02.937 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108256.log.gz 2024-04-25T04:46:02.937 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133384.log 2024-04-25T04:46:02.938 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97499.log.gz 2024-04-25T04:46:02.938 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147134.log 2024-04-25T04:46:02.939 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133384.log.gz 2024-04-25T04:46:02.939 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46782.log 2024-04-25T04:46:02.939 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147134.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147134.log.gz 2024-04-25T04:46:02.940 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111479.log 2024-04-25T04:46:02.940 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46782.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46782.log.gz 2024-04-25T04:46:02.941 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85313.log 2024-04-25T04:46:02.941 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111479.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111479.log.gz 2024-04-25T04:46:02.941 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116170.log 2024-04-25T04:46:02.942 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85313.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85313.log.gz 2024-04-25T04:46:02.942 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65117.log 2024-04-25T04:46:02.943 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116170.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116170.log.gz 2024-04-25T04:46:02.943 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141234.log 2024-04-25T04:46:02.944 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65117.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65117.log.gz 2024-04-25T04:46:02.944 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77284.log 2024-04-25T04:46:02.945 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141234.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141234.log.gz 2024-04-25T04:46:02.945 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120135.log 2024-04-25T04:46:02.946 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77284.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77284.log.gz 2024-04-25T04:46:02.946 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144941.log 2024-04-25T04:46:02.946 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120135.log.gz 2024-04-25T04:46:02.947 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45184.log 2024-04-25T04:46:02.947 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144941.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144941.log.gz 2024-04-25T04:46:02.947 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43718.log 2024-04-25T04:46:02.948 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45184.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45184.log.gz 2024-04-25T04:46:02.948 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125699.log 2024-04-25T04:46:02.949 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43718.log.gz 2024-04-25T04:46:02.949 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141590.log 2024-04-25T04:46:02.950 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125699.log.gz 2024-04-25T04:46:02.950 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72114.log 2024-04-25T04:46:02.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141590.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141590.log.gz 2024-04-25T04:46:02.951 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46591.log 2024-04-25T04:46:02.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72114.log.gz 2024-04-25T04:46:02.952 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115620.log 2024-04-25T04:46:02.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46591.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46591.log.gz 2024-04-25T04:46:02.953 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140794.log 2024-04-25T04:46:02.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115620.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115620.log.gz 2024-04-25T04:46:02.954 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137701.log 2024-04-25T04:46:02.954 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140794.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140794.log.gz 2024-04-25T04:46:02.954 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47842.log 2024-04-25T04:46:02.955 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137701.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137701.log.gz 2024-04-25T04:46:02.955 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89940.log 2024-04-25T04:46:02.956 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47842.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47842.log.gz 2024-04-25T04:46:02.956 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90075.log 2024-04-25T04:46:02.957 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89940.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89940.log.gz 2024-04-25T04:46:02.957 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134045.log 2024-04-25T04:46:02.958 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90075.log.gz 2024-04-25T04:46:02.958 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67985.log 2024-04-25T04:46:02.959 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134045.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134045.log.gz 2024-04-25T04:46:02.959 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146099.log 2024-04-25T04:46:02.960 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67985.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67985.log.gz 2024-04-25T04:46:02.960 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151870.log 2024-04-25T04:46:02.960 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146099.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146099.log.gz 2024-04-25T04:46:02.961 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122443.log 2024-04-25T04:46:02.962 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151870.log.gz 2024-04-25T04:46:02.962 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139119.log 2024-04-25T04:46:02.962 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122443.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122443.log.gz 2024-04-25T04:46:02.962 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91291.log 2024-04-25T04:46:02.963 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139119.log.gz 2024-04-25T04:46:02.963 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124913.log 2024-04-25T04:46:02.964 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91291.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91291.log.gz 2024-04-25T04:46:02.964 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85314.log 2024-04-25T04:46:02.965 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.124913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124913.log.gz 2024-04-25T04:46:02.965 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138805.log 2024-04-25T04:46:02.966 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85314.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85314.log.gz 2024-04-25T04:46:02.966 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128141.log 2024-04-25T04:46:02.967 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138805.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138805.log.gz 2024-04-25T04:46:02.967 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59979.log 2024-04-25T04:46:02.968 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128141.log.gz 2024-04-25T04:46:02.968 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124791.log 2024-04-25T04:46:02.969 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59979.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59979.log.gz 2024-04-25T04:46:02.969 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106104.log 2024-04-25T04:46:02.969 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.124791.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124791.log.gz 2024-04-25T04:46:02.970 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45793.log 2024-04-25T04:46:02.970 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106104.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106104.log.gz 2024-04-25T04:46:02.970 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89770.log 2024-04-25T04:46:02.971 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45793.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45793.log.gz 2024-04-25T04:46:02.971 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70343.log 2024-04-25T04:46:02.972 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89770.log.gz 2024-04-25T04:46:02.972 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97778.log 2024-04-25T04:46:02.973 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70343.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70343.log.gz 2024-04-25T04:46:02.973 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102241.log 2024-04-25T04:46:02.974 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97778.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97778.log.gz 2024-04-25T04:46:02.974 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116343.log 2024-04-25T04:46:02.975 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102241.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102241.log.gz 2024-04-25T04:46:02.975 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116832.log 2024-04-25T04:46:02.976 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116343.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116343.log.gz 2024-04-25T04:46:02.976 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138629.log 2024-04-25T04:46:02.976 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116832.log.gz 2024-04-25T04:46:02.977 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73671.log 2024-04-25T04:46:02.977 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138629.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138629.log.gz 2024-04-25T04:46:02.977 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108546.log 2024-04-25T04:46:02.978 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73671.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73671.log.gz 2024-04-25T04:46:02.978 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92335.log 2024-04-25T04:46:02.979 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108546.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108546.log.gz 2024-04-25T04:46:02.979 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97732.log 2024-04-25T04:46:02.980 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92335.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92335.log.gz 2024-04-25T04:46:02.980 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100535.log 2024-04-25T04:46:02.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97732.log.gz 2024-04-25T04:46:02.981 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128061.log 2024-04-25T04:46:02.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100535.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100535.log.gz 2024-04-25T04:46:02.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151036.log 2024-04-25T04:46:02.982 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128061.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128061.log.gz 2024-04-25T04:46:02.983 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70051.log 2024-04-25T04:46:02.983 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151036.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151036.log.gz 2024-04-25T04:46:02.984 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47769.log 2024-04-25T04:46:02.984 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70051.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70051.log.gz 2024-04-25T04:46:02.984 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143031.log 2024-04-25T04:46:02.985 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47769.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47769.log.gz 2024-04-25T04:46:02.985 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98630.log 2024-04-25T04:46:02.986 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143031.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143031.log.gz 2024-04-25T04:46:02.986 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54997.log 2024-04-25T04:46:02.987 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98630.log.gz 2024-04-25T04:46:02.987 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.149998.log 2024-04-25T04:46:02.988 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54997.log.gz 2024-04-25T04:46:02.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132339.log 2024-04-25T04:46:02.988 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.149998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.149998.log.gz 2024-04-25T04:46:02.989 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86445.log 2024-04-25T04:46:02.989 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132339.log.gz 2024-04-25T04:46:02.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148626.log 2024-04-25T04:46:02.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86445.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86445.log.gz 2024-04-25T04:46:02.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59298.log 2024-04-25T04:46:02.991 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148626.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148626.log.gz 2024-04-25T04:46:02.991 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66770.log 2024-04-25T04:46:02.992 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59298.log.gz 2024-04-25T04:46:02.992 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148357.log 2024-04-25T04:46:02.993 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66770.log.gz 2024-04-25T04:46:02.993 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.102347.log 2024-04-25T04:46:02.994 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148357.log.gz 2024-04-25T04:46:02.994 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101188.log 2024-04-25T04:46:02.995 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.102347.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.102347.log.gz 2024-04-25T04:46:02.995 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139537.log 2024-04-25T04:46:02.996 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101188.log.gz 2024-04-25T04:46:02.996 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153434.log 2024-04-25T04:46:02.997 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139537.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139537.log.gz 2024-04-25T04:46:02.997 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44462.log 2024-04-25T04:46:02.997 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153434.log.gz 2024-04-25T04:46:02.998 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133335.log 2024-04-25T04:46:02.998 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44462.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44462.log.gz 2024-04-25T04:46:02.998 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108697.log 2024-04-25T04:46:02.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133335.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133335.log.gz 2024-04-25T04:46:02.999 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45269.log 2024-04-25T04:46:03.000 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108697.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108697.log.gz 2024-04-25T04:46:03.000 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57276.log 2024-04-25T04:46:03.001 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45269.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45269.log.gz 2024-04-25T04:46:03.001 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73293.log 2024-04-25T04:46:03.002 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57276.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57276.log.gz 2024-04-25T04:46:03.002 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68196.log 2024-04-25T04:46:03.002 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73293.log.gz 2024-04-25T04:46:03.003 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91497.log 2024-04-25T04:46:03.003 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68196.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68196.log.gz 2024-04-25T04:46:03.004 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70972.log 2024-04-25T04:46:03.004 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91497.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91497.log.gz 2024-04-25T04:46:03.004 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48073.log 2024-04-25T04:46:03.005 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70972.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70972.log.gz 2024-04-25T04:46:03.006 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132035.log 2024-04-25T04:46:03.006 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48073.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48073.log.gz 2024-04-25T04:46:03.007 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139746.log 2024-04-25T04:46:03.007 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132035.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132035.log.gz 2024-04-25T04:46:03.007 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150658.log 2024-04-25T04:46:03.008 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139746.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139746.log.gz 2024-04-25T04:46:03.008 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104351.log 2024-04-25T04:46:03.009 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150658.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150658.log.gz 2024-04-25T04:46:03.009 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140960.log 2024-04-25T04:46:03.010 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104351.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104351.log.gz 2024-04-25T04:46:03.010 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86929.log 2024-04-25T04:46:03.011 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140960.log.gz 2024-04-25T04:46:03.011 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47965.log 2024-04-25T04:46:03.011 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86929.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86929.log.gz 2024-04-25T04:46:03.012 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154091.log 2024-04-25T04:46:03.012 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47965.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47965.log.gz 2024-04-25T04:46:03.013 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120429.log 2024-04-25T04:46:03.013 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154091.log.gz 2024-04-25T04:46:03.013 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113899.log 2024-04-25T04:46:03.014 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120429.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120429.log.gz 2024-04-25T04:46:03.014 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127036.log 2024-04-25T04:46:03.015 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113899.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113899.log.gz 2024-04-25T04:46:03.015 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76678.log 2024-04-25T04:46:03.016 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127036.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127036.log.gz 2024-04-25T04:46:03.016 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85679.log 2024-04-25T04:46:03.017 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76678.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76678.log.gz 2024-04-25T04:46:03.017 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148141.log 2024-04-25T04:46:03.017 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85679.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85679.log.gz 2024-04-25T04:46:03.018 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158900.log 2024-04-25T04:46:03.018 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148141.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148141.log.gz 2024-04-25T04:46:03.019 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63847.log 2024-04-25T04:46:03.019 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158900.log.gz 2024-04-25T04:46:03.020 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106482.log 2024-04-25T04:46:03.020 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63847.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63847.log.gz 2024-04-25T04:46:03.021 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107429.log 2024-04-25T04:46:03.021 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148398.log 0.0% 2024-04-25T04:46:03.021 INFO:teuthology.orchestra.run.smithi188.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.106482.log.gz 2024-04-25T04:46:03.022 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107429.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107429.log.gz 2024-04-25T04:46:03.022 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156651.log 2024-04-25T04:46:03.023 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148398.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148398.log.gz 2024-04-25T04:46:03.023 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159807.log 2024-04-25T04:46:03.024 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156651.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156651.log.gz 2024-04-25T04:46:03.024 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45122.log 2024-04-25T04:46:03.025 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.159807.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.159807.log.gz 2024-04-25T04:46:03.025 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64436.log 2024-04-25T04:46:03.025 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45122.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45122.log.gz 2024-04-25T04:46:03.026 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82681.log 2024-04-25T04:46:03.027 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64436.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64436.log.gz 2024-04-25T04:46:03.027 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61827.log 2024-04-25T04:46:03.027 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82681.log.gz 2024-04-25T04:46:03.028 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65785.log 2024-04-25T04:46:03.029 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61827.log.gz 2024-04-25T04:46:03.029 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51304.log 2024-04-25T04:46:03.029 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65785.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65785.log.gz 2024-04-25T04:46:03.030 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149938.log 2024-04-25T04:46:03.030 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51304.log.gz 2024-04-25T04:46:03.031 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120726.log 2024-04-25T04:46:03.032 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92249.log 2024-04-25T04:46:03.032 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149938.log.gz 2024-04-25T04:46:03.032 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120726.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120726.log.gz 2024-04-25T04:46:03.033 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117351.log 2024-04-25T04:46:03.033 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92249.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92249.log.gz 2024-04-25T04:46:03.034 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131334.log 2024-04-25T04:46:03.034 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117351.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117351.log.gz 2024-04-25T04:46:03.035 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103827.log 2024-04-25T04:46:03.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131334.log.gz 2024-04-25T04:46:03.035 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52002.log 2024-04-25T04:46:03.036 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103827.log.gz 2024-04-25T04:46:03.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77561.log 2024-04-25T04:46:03.037 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52002.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52002.log.gz 2024-04-25T04:46:03.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146615.log 2024-04-25T04:46:03.038 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77561.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77561.log.gz 2024-04-25T04:46:03.038 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99606.log 2024-04-25T04:46:03.039 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.146615.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146615.log.gz 2024-04-25T04:46:03.039 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53505.log 2024-04-25T04:46:03.040 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43319.log 2024-04-25T04:46:03.041 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99606.log.gz 2024-04-25T04:46:03.041 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53505.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53505.log.gz 2024-04-25T04:46:03.042 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63525.log 2024-04-25T04:46:03.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.43319.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43319.log.gz 2024-04-25T04:46:03.043 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76942.log 2024-04-25T04:46:03.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63525.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63525.log.gz 2024-04-25T04:46:03.044 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146955.log 2024-04-25T04:46:03.045 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76942.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76942.log.gz 2024-04-25T04:46:03.045 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64547.log 2024-04-25T04:46:03.045 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.146955.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146955.log.gz 2024-04-25T04:46:03.046 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113637.log 2024-04-25T04:46:03.047 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64547.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64547.log.gz 2024-04-25T04:46:03.047 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82683.log 2024-04-25T04:46:03.047 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113637.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113637.log.gz 2024-04-25T04:46:03.048 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89884.log 2024-04-25T04:46:03.049 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82683.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82683.log.gz 2024-04-25T04:46:03.049 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147377.log 2024-04-25T04:46:03.049 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89884.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89884.log.gz 2024-04-25T04:46:03.050 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97505.log 2024-04-25T04:46:03.051 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147377.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147377.log.gz 2024-04-25T04:46:03.051 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150268.log 2024-04-25T04:46:03.051 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97505.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97505.log.gz 2024-04-25T04:46:03.052 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101255.log 2024-04-25T04:46:03.053 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150268.log.gz 2024-04-25T04:46:03.053 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158215.log 2024-04-25T04:46:03.053 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101255.log.gz 2024-04-25T04:46:03.054 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110748.log 2024-04-25T04:46:03.055 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158215.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158215.log.gz 2024-04-25T04:46:03.055 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159282.log 2024-04-25T04:46:03.056 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110748.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110748.log.gz 2024-04-25T04:46:03.056 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103722.log 2024-04-25T04:46:03.057 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159282.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159282.log.gz 2024-04-25T04:46:03.057 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98910.log 2024-04-25T04:46:03.058 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103722.log.gz 2024-04-25T04:46:03.058 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60609.log 2024-04-25T04:46:03.059 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98910.log.gz 2024-04-25T04:46:03.059 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147021.log 2024-04-25T04:46:03.060 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60609.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60609.log.gz 2024-04-25T04:46:03.060 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119337.log 2024-04-25T04:46:03.061 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147021.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147021.log.gz 2024-04-25T04:46:03.061 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52845.log 2024-04-25T04:46:03.062 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119337.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119337.log.gz 2024-04-25T04:46:03.063 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92586.log 2024-04-25T04:46:03.063 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52845.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52845.log.gz 2024-04-25T04:46:03.064 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144801.log 2024-04-25T04:46:03.064 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92586.log.gz 2024-04-25T04:46:03.065 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83354.log 2024-04-25T04:46:03.066 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144801.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144801.log.gz 2024-04-25T04:46:03.066 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76723.log 2024-04-25T04:46:03.067 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83354.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83354.log.gz 2024-04-25T04:46:03.067 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77748.log 2024-04-25T04:46:03.067 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76723.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76723.log.gz 2024-04-25T04:46:03.068 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127541.log 2024-04-25T04:46:03.068 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77748.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77748.log.gz 2024-04-25T04:46:03.069 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139328.log 2024-04-25T04:46:03.070 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127541.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127541.log.gz 2024-04-25T04:46:03.070 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43212.log 2024-04-25T04:46:03.070 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139328.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139328.log.gz 2024-04-25T04:46:03.071 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105928.log 2024-04-25T04:46:03.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43212.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43212.log.gz 2024-04-25T04:46:03.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90830.log 2024-04-25T04:46:03.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105928.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105928.log.gz 2024-04-25T04:46:03.073 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116914.log 2024-04-25T04:46:03.074 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.90830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90830.log.gz 2024-04-25T04:46:03.074 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63905.log 2024-04-25T04:46:03.074 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116914.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116914.log.gz 2024-04-25T04:46:03.075 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90280.log 2024-04-25T04:46:03.076 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63905.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63905.log.gz 2024-04-25T04:46:03.076 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146797.log 2024-04-25T04:46:03.076 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90280.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90280.log.gz 2024-04-25T04:46:03.077 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86711.log 2024-04-25T04:46:03.078 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146797.log.gzgzip 2024-04-25T04:46:03.078 INFO:teuthology.orchestra.run.smithi188.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.71864.log 2024-04-25T04:46:03.078 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86711.log.gz 2024-04-25T04:46:03.079 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143363.log 2024-04-25T04:46:03.080 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71864.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71864.log.gz 2024-04-25T04:46:03.080 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117814.log 2024-04-25T04:46:03.081 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143363.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143363.log.gz 2024-04-25T04:46:03.081 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55617.log 2024-04-25T04:46:03.082 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117814.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117814.log.gz 2024-04-25T04:46:03.082 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68344.log 2024-04-25T04:46:03.083 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55617.log.gz 2024-04-25T04:46:03.083 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48666.log 2024-04-25T04:46:03.084 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68344.log.gz 2024-04-25T04:46:03.084 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82357.log 2024-04-25T04:46:03.085 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48666.log.gz 2024-04-25T04:46:03.085 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120008.log 2024-04-25T04:46:03.086 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82357.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82357.log.gz 2024-04-25T04:46:03.087 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81819.log 2024-04-25T04:46:03.087 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120008.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120008.log.gz 2024-04-25T04:46:03.088 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151431.log 2024-04-25T04:46:03.089 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81819.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81819.log.gz 2024-04-25T04:46:03.089 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152827.log 2024-04-25T04:46:03.089 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151431.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151431.log.gz 2024-04-25T04:46:03.090 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136322.log 2024-04-25T04:46:03.091 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152827.log.gz 2024-04-25T04:46:03.091 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64549.log 2024-04-25T04:46:03.091 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136322.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136322.log.gz 2024-04-25T04:46:03.092 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.102678.log 2024-04-25T04:46:03.092 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64549.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64549.log.gz 2024-04-25T04:46:03.093 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119966.log 2024-04-25T04:46:03.093 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.102678.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.102678.log.gz 2024-04-25T04:46:03.094 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74521.log 2024-04-25T04:46:03.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119966.log.gz 2024-04-25T04:46:03.095 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139810.log 2024-04-25T04:46:03.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74521.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74521.log.gz 2024-04-25T04:46:03.096 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92762.log 2024-04-25T04:46:03.097 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139810.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139810.log.gz 2024-04-25T04:46:03.097 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154006.log 2024-04-25T04:46:03.098 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92762.log.gz 2024-04-25T04:46:03.098 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132485.log 2024-04-25T04:46:03.099 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154006.log.gz 2024-04-25T04:46:03.099 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88741.log 2024-04-25T04:46:03.100 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132485.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132485.log.gz 2024-04-25T04:46:03.100 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50564.log 2024-04-25T04:46:03.101 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88741.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88741.log.gz 2024-04-25T04:46:03.101 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73542.log 2024-04-25T04:46:03.102 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50564.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50564.log.gz 2024-04-25T04:46:03.102 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149627.log 2024-04-25T04:46:03.103 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73542.log.gz 2024-04-25T04:46:03.103 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122721.log 2024-04-25T04:46:03.104 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149627.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149627.log.gz 2024-04-25T04:46:03.104 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123204.log 2024-04-25T04:46:03.105 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122721.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122721.log.gz 2024-04-25T04:46:03.105 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115187.log 2024-04-25T04:46:03.106 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123204.log.gz 2024-04-25T04:46:03.106 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100539.log 2024-04-25T04:46:03.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115187.log.gz 2024-04-25T04:46:03.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113429.log 2024-04-25T04:46:03.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100539.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100539.log.gz 2024-04-25T04:46:03.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129139.log 2024-04-25T04:46:03.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113429.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113429.log.gz 2024-04-25T04:46:03.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93118.log 2024-04-25T04:46:03.109 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129139.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129139.log.gz 2024-04-25T04:46:03.109 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159189.log 2024-04-25T04:46:03.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93118.log.gz 2024-04-25T04:46:03.110 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156136.log 2024-04-25T04:46:03.111 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159189.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159189.log.gz 2024-04-25T04:46:03.111 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42011.log 2024-04-25T04:46:03.112 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156136.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156136.log.gz 2024-04-25T04:46:03.112 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65717.log 2024-04-25T04:46:03.113 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.42011.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42011.log.gz 2024-04-25T04:46:03.113 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48325.log 2024-04-25T04:46:03.114 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65717.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65717.log.gz 2024-04-25T04:46:03.114 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158437.log 2024-04-25T04:46:03.114 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48325.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48325.log.gz 2024-04-25T04:46:03.115 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130882.log 2024-04-25T04:46:03.115 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158437.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158437.log.gz 2024-04-25T04:46:03.116 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36521.log 2024-04-25T04:46:03.116 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130882.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130882.log.gz 2024-04-25T04:46:03.116 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85652.log 2024-04-25T04:46:03.117 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.36521.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36521.log.gz 2024-04-25T04:46:03.117 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153732.log 2024-04-25T04:46:03.118 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85652.log.gz 2024-04-25T04:46:03.119 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71103.log 2024-04-25T04:46:03.119 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153732.log.gz 2024-04-25T04:46:03.119 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140584.log 2024-04-25T04:46:03.120 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71103.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71103.log.gz 2024-04-25T04:46:03.120 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110078.log 2024-04-25T04:46:03.121 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140584.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140584.log.gz 2024-04-25T04:46:03.121 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143535.log 2024-04-25T04:46:03.122 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110078.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110078.log.gz 2024-04-25T04:46:03.122 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73756.log 2024-04-25T04:46:03.123 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143535.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143535.log.gz 2024-04-25T04:46:03.123 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127158.log 2024-04-25T04:46:03.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73756.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73756.log.gz 2024-04-25T04:46:03.124 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65624.log 2024-04-25T04:46:03.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127158.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127158.log.gz 2024-04-25T04:46:03.124 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41719.log 2024-04-25T04:46:03.125 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.65624.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65624.log.gz 2024-04-25T04:46:03.125 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68046.log 2024-04-25T04:46:03.126 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41719.log.gz 2024-04-25T04:46:03.126 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85729.log 2024-04-25T04:46:03.127 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68046.log.gz 2024-04-25T04:46:03.127 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143618.log 2024-04-25T04:46:03.128 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85729.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85729.log.gz 2024-04-25T04:46:03.128 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152970.log 2024-04-25T04:46:03.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143618.log.gz 2024-04-25T04:46:03.129 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138188.log 2024-04-25T04:46:03.130 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152970.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152970.log.gz 2024-04-25T04:46:03.130 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54104.log 2024-04-25T04:46:03.131 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138188.log.gz 2024-04-25T04:46:03.131 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147215.log 2024-04-25T04:46:03.131 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54104.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54104.log.gz 2024-04-25T04:46:03.132 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134276.log 2024-04-25T04:46:03.132 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147215.log.gz 2024-04-25T04:46:03.132 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150265.log 2024-04-25T04:46:03.133 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134276.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134276.log.gz 2024-04-25T04:46:03.133 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124583.log 2024-04-25T04:46:03.134 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150265.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150265.log.gz 2024-04-25T04:46:03.134 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54633.log 2024-04-25T04:46:03.135 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124583.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124583.log.gz 2024-04-25T04:46:03.135 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125214.log 2024-04-25T04:46:03.136 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54633.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54633.log.gz 2024-04-25T04:46:03.136 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46153.log 2024-04-25T04:46:03.137 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125214.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125214.log.gz 2024-04-25T04:46:03.137 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152303.log 2024-04-25T04:46:03.138 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46153.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46153.log.gz 2024-04-25T04:46:03.138 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138067.log 2024-04-25T04:46:03.138 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152303.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152303.log.gz 2024-04-25T04:46:03.139 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88701.log 2024-04-25T04:46:03.139 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.138067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138067.log.gz 2024-04-25T04:46:03.140 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59895.log 2024-04-25T04:46:03.140 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88701.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88701.log.gz 2024-04-25T04:46:03.140 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139288.log 2024-04-25T04:46:03.141 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59895.log.gz 2024-04-25T04:46:03.141 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147022.log 2024-04-25T04:46:03.142 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139288.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139288.log.gz 2024-04-25T04:46:03.142 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122650.log 2024-04-25T04:46:03.143 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147022.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147022.log.gz 2024-04-25T04:46:03.143 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73292.log 2024-04-25T04:46:03.144 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122650.log.gz 2024-04-25T04:46:03.144 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81732.log 2024-04-25T04:46:03.145 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73292.log.gz 2024-04-25T04:46:03.145 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84203.log 2024-04-25T04:46:03.146 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81732.log.gz 2024-04-25T04:46:03.146 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141802.log 2024-04-25T04:46:03.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84203.log.gz 2024-04-25T04:46:03.147 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144744.log 2024-04-25T04:46:03.148 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141802.log.gz 2024-04-25T04:46:03.148 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96005.log 2024-04-25T04:46:03.148 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144744.log.gz 2024-04-25T04:46:03.148 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86687.log 2024-04-25T04:46:03.149 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96005.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96005.log.gz 2024-04-25T04:46:03.149 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128379.log 2024-04-25T04:46:03.150 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86687.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86687.log.gz 2024-04-25T04:46:03.150 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90568.log 2024-04-25T04:46:03.151 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128379.log.gz 2024-04-25T04:46:03.151 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156618.log 2024-04-25T04:46:03.152 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90568.log.gz 2024-04-25T04:46:03.152 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77231.log 2024-04-25T04:46:03.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156618.log.gz 2024-04-25T04:46:03.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150405.log 2024-04-25T04:46:03.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77231.log.gz 2024-04-25T04:46:03.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97584.log 2024-04-25T04:46:03.155 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150405.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.150405.log.gz/var/log/ceph/cluster2-client.mirror.89635.log 2024-04-25T04:46:03.155 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-25T04:46:03.155 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97584.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97584.log.gz 2024-04-25T04:46:03.156 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77211.log 2024-04-25T04:46:03.156 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89635.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89635.log.gz 2024-04-25T04:46:03.156 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144293.log 2024-04-25T04:46:03.157 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77211.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77211.log.gz 2024-04-25T04:46:03.157 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119673.log 2024-04-25T04:46:03.158 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144293.log.gz 2024-04-25T04:46:03.158 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87308.log 2024-04-25T04:46:03.159 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119673.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119673.log.gz 2024-04-25T04:46:03.159 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153055.log 2024-04-25T04:46:03.160 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87308.log.gz 2024-04-25T04:46:03.160 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153960.log 2024-04-25T04:46:03.161 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153055.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153055.log.gz 2024-04-25T04:46:03.161 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133082.log 2024-04-25T04:46:03.161 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153960.log.gz 2024-04-25T04:46:03.162 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49840.log 2024-04-25T04:46:03.162 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133082.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133082.log.gz 2024-04-25T04:46:03.162 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108018.log 2024-04-25T04:46:03.164 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49840.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49840.log.gz 2024-04-25T04:46:03.164 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65925.log 2024-04-25T04:46:03.164 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108018.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108018.log.gz 2024-04-25T04:46:03.164 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138698.log 2024-04-25T04:46:03.165 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65925.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65925.log.gz 2024-04-25T04:46:03.165 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146274.log 2024-04-25T04:46:03.166 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138698.log.gz 2024-04-25T04:46:03.166 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119715.log 2024-04-25T04:46:03.167 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146274.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146274.log.gz 2024-04-25T04:46:03.167 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138699.log 2024-04-25T04:46:03.168 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119715.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119715.log.gz 2024-04-25T04:46:03.168 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88897.log 2024-04-25T04:46:03.169 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138699.log.gz 2024-04-25T04:46:03.169 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141716.log 2024-04-25T04:46:03.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.88897.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88897.log.gz 2024-04-25T04:46:03.170 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96277.log 2024-04-25T04:46:03.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141716.log.gz 2024-04-25T04:46:03.171 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69032.log 2024-04-25T04:46:03.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96277.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96277.log.gz 2024-04-25T04:46:03.172 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107243.log 2024-04-25T04:46:03.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69032.log.gz 2024-04-25T04:46:03.172 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82070.log 2024-04-25T04:46:03.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107243.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107243.log.gz 2024-04-25T04:46:03.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147711.log 2024-04-25T04:46:03.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82070.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82070.log.gz 2024-04-25T04:46:03.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115753.log 2024-04-25T04:46:03.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147711.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147711.log.gz 2024-04-25T04:46:03.175 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57530.log 2024-04-25T04:46:03.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115753.log.gz 2024-04-25T04:46:03.176 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42748.log 2024-04-25T04:46:03.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57530.log.gz 2024-04-25T04:46:03.177 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141024.log 2024-04-25T04:46:03.177 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42748.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42748.log.gz 2024-04-25T04:46:03.178 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101141.log 2024-04-25T04:46:03.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141024.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141024.log.gz 2024-04-25T04:46:03.178 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115982.log 2024-04-25T04:46:03.179 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101141.log.gz 2024-04-25T04:46:03.179 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104559.log 2024-04-25T04:46:03.180 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115982.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115982.log.gz 2024-04-25T04:46:03.180 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131384.log 2024-04-25T04:46:03.181 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104559.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104559.log.gz 2024-04-25T04:46:03.181 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88809.log 2024-04-25T04:46:03.182 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131384.log.gz 2024-04-25T04:46:03.182 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86291.log 2024-04-25T04:46:03.183 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88809.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88809.log.gz 2024-04-25T04:46:03.183 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74500.log 2024-04-25T04:46:03.184 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86291.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86291.log.gz 2024-04-25T04:46:03.184 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97015.log 2024-04-25T04:46:03.184 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74500.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74500.log.gz 2024-04-25T04:46:03.185 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107090.log 2024-04-25T04:46:03.186 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97015.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97015.log.gz 2024-04-25T04:46:03.186 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85707.log 2024-04-25T04:46:03.187 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107090.log.gz 2024-04-25T04:46:03.187 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132386.log 2024-04-25T04:46:03.187 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85707.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85707.log.gz 2024-04-25T04:46:03.188 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54595.log 2024-04-25T04:46:03.188 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132386.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132386.log.gz 2024-04-25T04:46:03.189 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64097.log 2024-04-25T04:46:03.189 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54595.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54595.log.gz 2024-04-25T04:46:03.189 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48157.log 2024-04-25T04:46:03.190 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64097.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64097.log.gz 2024-04-25T04:46:03.190 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108210.log 2024-04-25T04:46:03.191 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48157.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48157.log.gz 2024-04-25T04:46:03.191 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97814.log 2024-04-25T04:46:03.192 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108210.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108210.log.gz 2024-04-25T04:46:03.192 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136672.log 2024-04-25T04:46:03.193 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97814.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97814.log.gz 2024-04-25T04:46:03.193 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143847.log 2024-04-25T04:46:03.193 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136672.log.gz 2024-04-25T04:46:03.194 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120010.log 2024-04-25T04:46:03.194 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143847.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143847.log.gz 2024-04-25T04:46:03.195 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111780.log 2024-04-25T04:46:03.195 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120010.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120010.log.gz 2024-04-25T04:46:03.196 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157682.log 2024-04-25T04:46:03.196 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.111780.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111780.log.gz 2024-04-25T04:46:03.196 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128910.log 2024-04-25T04:46:03.197 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157682.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157682.log.gz 2024-04-25T04:46:03.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43670.log 2024-04-25T04:46:03.198 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128910.log.gz 2024-04-25T04:46:03.198 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89086.log 2024-04-25T04:46:03.199 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.43670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43670.log.gz 2024-04-25T04:46:03.199 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156501.log 2024-04-25T04:46:03.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89086.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89086.log.gz 2024-04-25T04:46:03.200 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151725.log 2024-04-25T04:46:03.201 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156501.log.gz 2024-04-25T04:46:03.201 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155274.log 2024-04-25T04:46:03.201 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151725.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151725.log.gz 2024-04-25T04:46:03.202 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105261.log 2024-04-25T04:46:03.202 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155274.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155274.log.gz 2024-04-25T04:46:03.203 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66194.log 2024-04-25T04:46:03.203 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105261.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105261.log.gz 2024-04-25T04:46:03.203 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151643.log 2024-04-25T04:46:03.204 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66194.log.gz 2024-04-25T04:46:03.204 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109579.log 2024-04-25T04:46:03.205 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151643.log.gz 2024-04-25T04:46:03.205 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76889.log 2024-04-25T04:46:03.206 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109579.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109579.log.gz 2024-04-25T04:46:03.206 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105408.log 2024-04-25T04:46:03.207 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.76889.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76889.log.gz 2024-04-25T04:46:03.207 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108396.log 2024-04-25T04:46:03.208 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105408.log.gz 2024-04-25T04:46:03.208 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134815.log 2024-04-25T04:46:03.209 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108396.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108396.log.gz 2024-04-25T04:46:03.209 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137970.log 2024-04-25T04:46:03.209 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134815.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134815.log.gz 2024-04-25T04:46:03.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126719.log 2024-04-25T04:46:03.210 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137970.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137970.log.gz 2024-04-25T04:46:03.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118384.log 2024-04-25T04:46:03.211 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126719.log.gz 2024-04-25T04:46:03.211 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140249.log 2024-04-25T04:46:03.212 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118384.log.gz 2024-04-25T04:46:03.212 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75656.log 2024-04-25T04:46:03.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140249.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140249.log.gz 2024-04-25T04:46:03.213 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112831.log 2024-04-25T04:46:03.214 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75656.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75656.log.gz 2024-04-25T04:46:03.214 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87447.log 2024-04-25T04:46:03.214 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.112831.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112831.log.gz 2024-04-25T04:46:03.215 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134112.log 2024-04-25T04:46:03.215 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87447.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87447.log.gz 2024-04-25T04:46:03.216 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77789.log 2024-04-25T04:46:03.216 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134112.log.gz 2024-04-25T04:46:03.217 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94105.log 2024-04-25T04:46:03.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77789.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77789.log.gz 2024-04-25T04:46:03.218 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158554.log 2024-04-25T04:46:03.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94105.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94105.log.gz 2024-04-25T04:46:03.218 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70722.log 2024-04-25T04:46:03.219 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158554.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158554.log.gz 2024-04-25T04:46:03.219 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125188.log 2024-04-25T04:46:03.220 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70722.log.gz 2024-04-25T04:46:03.220 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54172.log 2024-04-25T04:46:03.221 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125188.log.gz 2024-04-25T04:46:03.221 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117479.log 2024-04-25T04:46:03.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54172.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54172.log.gz 2024-04-25T04:46:03.222 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104202.log 2024-04-25T04:46:03.223 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117479.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117479.log.gz 2024-04-25T04:46:03.223 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96570.log 2024-04-25T04:46:03.224 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104202.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104202.log.gz 2024-04-25T04:46:03.224 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45835.log 2024-04-25T04:46:03.224 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96570.log.gz 2024-04-25T04:46:03.225 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138517.log 2024-04-25T04:46:03.226 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45835.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45835.log.gz 2024-04-25T04:46:03.226 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84538.log 2024-04-25T04:46:03.227 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138517.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138517.log.gz 2024-04-25T04:46:03.227 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51524.log 2024-04-25T04:46:03.228 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84538.log.gz 2024-04-25T04:46:03.228 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84055.log 2024-04-25T04:46:03.229 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51524.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51524.log.gz 2024-04-25T04:46:03.229 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57192.log 2024-04-25T04:46:03.230 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84055.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84055.log.gz 2024-04-25T04:46:03.230 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149754.log 2024-04-25T04:46:03.231 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57192.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57192.log.gz 2024-04-25T04:46:03.231 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85815.log 2024-04-25T04:46:03.231 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149754.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149754.log.gz 2024-04-25T04:46:03.232 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124312.log 2024-04-25T04:46:03.232 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85815.log.gz 2024-04-25T04:46:03.233 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62939.log 2024-04-25T04:46:03.233 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124312.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124312.log.gz 2024-04-25T04:46:03.234 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102304.log 2024-04-25T04:46:03.235 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62939.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62939.log.gz 2024-04-25T04:46:03.235 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124043.log 2024-04-25T04:46:03.235 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102304.log.gz 2024-04-25T04:46:03.235 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51616.log 2024-04-25T04:46:03.236 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124043.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124043.log.gz 2024-04-25T04:46:03.236 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134177.log 2024-04-25T04:46:03.237 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51616.log.gz 2024-04-25T04:46:03.237 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135980.log 2024-04-25T04:46:03.238 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134177.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134177.log.gz 2024-04-25T04:46:03.238 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91122.log 2024-04-25T04:46:03.239 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135980.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135980.log.gz 2024-04-25T04:46:03.239 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100583.log 2024-04-25T04:46:03.240 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.91122.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91122.log.gz 2024-04-25T04:46:03.240 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58209.log 2024-04-25T04:46:03.241 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100583.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100583.log.gz 2024-04-25T04:46:03.241 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133913.log 2024-04-25T04:46:03.242 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58209.log.gz 2024-04-25T04:46:03.242 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56800.log 2024-04-25T04:46:03.242 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133913.log.gz 2024-04-25T04:46:03.243 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62896.log 2024-04-25T04:46:03.243 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56800.log.gz 2024-04-25T04:46:03.244 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139183.log 2024-04-25T04:46:03.244 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62896.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62896.log.gz 2024-04-25T04:46:03.244 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117139.log 2024-04-25T04:46:03.245 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139183.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139183.log.gz 2024-04-25T04:46:03.245 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81336.log 2024-04-25T04:46:03.246 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117139.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117139.log.gz 2024-04-25T04:46:03.246 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139599.log 2024-04-25T04:46:03.247 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81336.log.gz 2024-04-25T04:46:03.247 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128953.log 2024-04-25T04:46:03.248 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139599.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139599.log.gz 2024-04-25T04:46:03.248 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98674.log 2024-04-25T04:46:03.248 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128953.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128953.log.gz 2024-04-25T04:46:03.249 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89656.log 2024-04-25T04:46:03.250 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98674.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98674.log.gz 2024-04-25T04:46:03.250 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69293.log 2024-04-25T04:46:03.251 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89656.log.gz 2024-04-25T04:46:03.251 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76147.log 2024-04-25T04:46:03.251 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69293.log.gz 2024-04-25T04:46:03.251 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65274.log 2024-04-25T04:46:03.252 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76147.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76147.log.gz 2024-04-25T04:46:03.252 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153893.log 2024-04-25T04:46:03.253 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.65274.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65274.log.gz 2024-04-25T04:46:03.253 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89703.log 2024-04-25T04:46:03.254 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153893.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153893.log.gz 2024-04-25T04:46:03.254 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150447.log 2024-04-25T04:46:03.255 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89703.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89703.log.gz 2024-04-25T04:46:03.255 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116771.log 2024-04-25T04:46:03.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150447.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150447.log.gz 2024-04-25T04:46:03.256 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102608.log 2024-04-25T04:46:03.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116771.log.gz 2024-04-25T04:46:03.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75528.log 2024-04-25T04:46:03.257 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102608.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102608.log.gz 2024-04-25T04:46:03.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56501.log 2024-04-25T04:46:03.258 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75528.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75528.log.gz 2024-04-25T04:46:03.258 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135903.log 2024-04-25T04:46:03.259 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56501.log.gz 2024-04-25T04:46:03.259 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145764.log 2024-04-25T04:46:03.260 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135903.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135903.log.gz 2024-04-25T04:46:03.262 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157681.log 2024-04-25T04:46:03.262 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145764.log.gz 2024-04-25T04:46:03.262 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131646.log 2024-04-25T04:46:03.262 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157681.log.gz 2024-04-25T04:46:03.262 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43368.log 2024-04-25T04:46:03.263 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131646.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131646.log.gz 2024-04-25T04:46:03.263 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66476.log 2024-04-25T04:46:03.263 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43368.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43368.log.gz 2024-04-25T04:46:03.264 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54058.log 2024-04-25T04:46:03.264 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66476.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66476.log.gz 2024-04-25T04:46:03.264 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91834.log 2024-04-25T04:46:03.265 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54058.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54058.log.gz 2024-04-25T04:46:03.265 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128103.log 2024-04-25T04:46:03.266 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91834.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91834.log.gz 2024-04-25T04:46:03.266 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58987.log 2024-04-25T04:46:03.267 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128103.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128103.log.gz 2024-04-25T04:46:03.267 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61363.log 2024-04-25T04:46:03.268 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58987.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58987.log.gz 2024-04-25T04:46:03.268 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148982.log 2024-04-25T04:46:03.268 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61363.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61363.log.gz 2024-04-25T04:46:03.269 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60188.log 2024-04-25T04:46:03.269 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148982.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148982.log.gz 2024-04-25T04:46:03.269 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132660.log 2024-04-25T04:46:03.270 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60188.log.gz 2024-04-25T04:46:03.270 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93075.log 2024-04-25T04:46:03.271 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132660.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132660.log.gz 2024-04-25T04:46:03.271 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81986.log 2024-04-25T04:46:03.272 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93075.log.gz 2024-04-25T04:46:03.272 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56565.log 2024-04-25T04:46:03.273 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81986.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81986.log.gz 2024-04-25T04:46:03.273 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158665.log 2024-04-25T04:46:03.274 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56565.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56565.log.gz 2024-04-25T04:46:03.274 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54730.log 2024-04-25T04:46:03.275 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158665.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158665.log.gz 2024-04-25T04:46:03.275 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60734.log 2024-04-25T04:46:03.275 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54730.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54730.log.gz 2024-04-25T04:46:03.276 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65510.log 2024-04-25T04:46:03.276 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60734.log.gz 2024-04-25T04:46:03.277 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139515.log 2024-04-25T04:46:03.277 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.65510.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65510.log.gz 2024-04-25T04:46:03.278 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134945.log 2024-04-25T04:46:03.278 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139515.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139515.log.gz 2024-04-25T04:46:03.278 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128700.log 2024-04-25T04:46:03.279 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134945.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134945.log.gz 2024-04-25T04:46:03.279 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105850.log 2024-04-25T04:46:03.280 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128700.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128700.log.gz 2024-04-25T04:46:03.280 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147295.log 2024-04-25T04:46:03.281 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105850.log.gz 2024-04-25T04:46:03.281 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63160.log 2024-04-25T04:46:03.282 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.147295.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147295.log.gz 2024-04-25T04:46:03.282 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105095.log 2024-04-25T04:46:03.283 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63160.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63160.log.gz 2024-04-25T04:46:03.283 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136927.log 2024-04-25T04:46:03.283 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105095.log.gz 2024-04-25T04:46:03.284 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141152.log 2024-04-25T04:46:03.284 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136927.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136927.log.gz 2024-04-25T04:46:03.285 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126723.log 2024-04-25T04:46:03.285 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141152.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141152.log.gz 2024-04-25T04:46:03.285 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68612.log 2024-04-25T04:46:03.286 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126723.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126723.log.gz 2024-04-25T04:46:03.287 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48998.log 2024-04-25T04:46:03.287 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68612.log.gz 2024-04-25T04:46:03.287 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92271.log 2024-04-25T04:46:03.288 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48998.log.gz 2024-04-25T04:46:03.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80279.log 2024-04-25T04:46:03.289 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92271.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92271.log.gz 2024-04-25T04:46:03.289 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114654.log 2024-04-25T04:46:03.290 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80279.log.gz 2024-04-25T04:46:03.290 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117652.log 2024-04-25T04:46:03.290 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114654.log.gz 2024-04-25T04:46:03.291 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99414.log 2024-04-25T04:46:03.291 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117652.log.gz 2024-04-25T04:46:03.292 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74735.log 2024-04-25T04:46:03.292 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99414.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99414.log.gz 2024-04-25T04:46:03.293 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142733.log 2024-04-25T04:46:03.293 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74735.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74735.log.gz 2024-04-25T04:46:03.294 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103616.log 2024-04-25T04:46:03.294 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142733.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142733.log.gz 2024-04-25T04:46:03.294 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140795.log 2024-04-25T04:46:03.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103616.log.gz 2024-04-25T04:46:03.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103891.log 2024-04-25T04:46:03.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140795.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140795.log.gz 2024-04-25T04:46:03.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71266.log 2024-04-25T04:46:03.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103891.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103891.log.gz 2024-04-25T04:46:03.297 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41328.log 2024-04-25T04:46:03.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71266.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71266.log.gz 2024-04-25T04:46:03.298 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134492.log 2024-04-25T04:46:03.298 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.41328.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41328.log.gz 2024-04-25T04:46:03.299 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160592.log 2024-04-25T04:46:03.299 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134492.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134492.log.gz 2024-04-25T04:46:03.299 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49457.log 2024-04-25T04:46:03.300 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160592.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160592.log.gz 2024-04-25T04:46:03.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153978.log 2024-04-25T04:46:03.301 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49457.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49457.log.gz 2024-04-25T04:46:03.301 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72952.log 2024-04-25T04:46:03.302 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153978.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153978.log.gz 2024-04-25T04:46:03.302 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69946.log 2024-04-25T04:46:03.303 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72952.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72952.log.gz 2024-04-25T04:46:03.303 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80049.log 2024-04-25T04:46:03.304 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69946.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69946.log.gz 2024-04-25T04:46:03.304 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48179.log 2024-04-25T04:46:03.304 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80049.log.gz 2024-04-25T04:46:03.305 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160230.log 2024-04-25T04:46:03.305 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48179.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48179.log.gz 2024-04-25T04:46:03.305 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96004.log 2024-04-25T04:46:03.306 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160230.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160230.log.gz 2024-04-25T04:46:03.307 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91917.log 2024-04-25T04:46:03.307 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96004.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96004.log.gz 2024-04-25T04:46:03.307 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43120.log 2024-04-25T04:46:03.308 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91917.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91917.log.gz 2024-04-25T04:46:03.308 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96593.log 2024-04-25T04:46:03.309 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43120.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43120.log.gz 2024-04-25T04:46:03.309 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94114.log 2024-04-25T04:46:03.310 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96593.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96593.log.gz 2024-04-25T04:46:03.310 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133310.log 2024-04-25T04:46:03.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94114.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94114.log.gz 2024-04-25T04:46:03.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53593.log 2024-04-25T04:46:03.312 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133310.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133310.log.gz 2024-04-25T04:46:03.312 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119038.log 2024-04-25T04:46:03.313 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59197.log 2024-04-25T04:46:03.313 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53593.log.gz 2024-04-25T04:46:03.313 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119038.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119038.log.gz 2024-04-25T04:46:03.314 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49376.log 2024-04-25T04:46:03.314 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59197.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59197.log.gz 2024-04-25T04:46:03.315 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130996.log 2024-04-25T04:46:03.315 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49376.log.gz 2024-04-25T04:46:03.316 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124933.log 2024-04-25T04:46:03.316 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130996.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130996.log.gz 2024-04-25T04:46:03.317 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147674.log 2024-04-25T04:46:03.317 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124933.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124933.log.gz 2024-04-25T04:46:03.317 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111593.log 2024-04-25T04:46:03.318 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147674.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147674.log.gz 2024-04-25T04:46:03.318 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87737.log 2024-04-25T04:46:03.319 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.111593.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111593.log.gz 2024-04-25T04:46:03.319 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54767.log 2024-04-25T04:46:03.320 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87737.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87737.log.gz 2024-04-25T04:46:03.320 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79233.log 2024-04-25T04:46:03.320 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54767.log.gz 2024-04-25T04:46:03.321 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153374.log 2024-04-25T04:46:03.321 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79233.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79233.log.gz 2024-04-25T04:46:03.322 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103514.log 2024-04-25T04:46:03.322 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153374.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153374.log.gz 2024-04-25T04:46:03.322 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90790.log 2024-04-25T04:46:03.323 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103514.log.gz 2024-04-25T04:46:03.323 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116024.log 2024-04-25T04:46:03.324 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90790.log.gz 2024-04-25T04:46:03.324 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74734.log 2024-04-25T04:46:03.325 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116024.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116024.log.gz 2024-04-25T04:46:03.325 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127330.log 2024-04-25T04:46:03.326 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74734.log.gz 2024-04-25T04:46:03.326 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133781.log 2024-04-25T04:46:03.326 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127330.log.gz 2024-04-25T04:46:03.327 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36424.log 2024-04-25T04:46:03.327 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133781.log.gz 2024-04-25T04:46:03.328 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112082.log 2024-04-25T04:46:03.328 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.36424.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36424.log.gz 2024-04-25T04:46:03.328 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144935.log 2024-04-25T04:46:03.329 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112082.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112082.log.gz 2024-04-25T04:46:03.329 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151118.log 2024-04-25T04:46:03.330 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144935.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144935.log.gz 2024-04-25T04:46:03.330 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89962.log 2024-04-25T04:46:03.331 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151118.log.gz 2024-04-25T04:46:03.331 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122267.log 2024-04-25T04:46:03.332 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89962.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89962.log.gz 2024-04-25T04:46:03.332 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143495.log 2024-04-25T04:46:03.333 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122267.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122267.log.gz 2024-04-25T04:46:03.333 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69372.log 2024-04-25T04:46:03.333 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143495.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143495.log.gz 2024-04-25T04:46:03.334 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42809.log 2024-04-25T04:46:03.334 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69372.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69372.log.gz 2024-04-25T04:46:03.335 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133312.log 2024-04-25T04:46:03.335 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141551.log 0.0% 2024-04-25T04:46:03.335 INFO:teuthology.orchestra.run.smithi188.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42809.log.gz 2024-04-25T04:46:03.336 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133312.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133312.log.gz 2024-04-25T04:46:03.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149118.log 2024-04-25T04:46:03.337 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141551.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141551.log.gz 2024-04-25T04:46:03.337 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153373.log 2024-04-25T04:46:03.338 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149118.log.gz 2024-04-25T04:46:03.338 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105475.log 2024-04-25T04:46:03.339 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153373.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153373.log.gz 2024-04-25T04:46:03.339 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93034.log 2024-04-25T04:46:03.339 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105475.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105475.log.gz 2024-04-25T04:46:03.340 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70136.log 2024-04-25T04:46:03.340 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93034.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93034.log.gz 2024-04-25T04:46:03.340 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82427.log 2024-04-25T04:46:03.341 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70136.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70136.log.gz 2024-04-25T04:46:03.341 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57600.log 2024-04-25T04:46:03.342 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82427.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82427.log.gz 2024-04-25T04:46:03.342 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55994.log 2024-04-25T04:46:03.343 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57600.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57600.log.gz 2024-04-25T04:46:03.343 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145587.log 2024-04-25T04:46:03.344 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55994.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55994.log.gz 2024-04-25T04:46:03.344 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131856.log 2024-04-25T04:46:03.345 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145587.log.gz 2024-04-25T04:46:03.345 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73712.log 2024-04-25T04:46:03.345 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131856.log.gz 2024-04-25T04:46:03.346 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151986.log 2024-04-25T04:46:03.346 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73712.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73712.log.gz 2024-04-25T04:46:03.346 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42235.log 2024-04-25T04:46:03.347 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151986.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151986.log.gz 2024-04-25T04:46:03.347 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129438.log 2024-04-25T04:46:03.348 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42235.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42235.log.gz 2024-04-25T04:46:03.348 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102397.log 2024-04-25T04:46:03.349 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129438.log.gz 2024-04-25T04:46:03.349 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156047.log 2024-04-25T04:46:03.350 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102397.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102397.log.gz 2024-04-25T04:46:03.350 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141718.log 2024-04-25T04:46:03.350 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156047.log.gz 2024-04-25T04:46:03.351 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119674.log 2024-04-25T04:46:03.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141718.log.gz 2024-04-25T04:46:03.352 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78610.log 2024-04-25T04:46:03.352 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119674.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119674.log.gz 2024-04-25T04:46:03.353 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109094.log 2024-04-25T04:46:03.354 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78610.log.gz 2024-04-25T04:46:03.354 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128335.log 2024-04-25T04:46:03.354 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109094.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109094.log.gz 2024-04-25T04:46:03.355 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71102.log 2024-04-25T04:46:03.355 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128335.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128335.log.gz 2024-04-25T04:46:03.355 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130968.log 2024-04-25T04:46:03.356 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71102.log.gz 2024-04-25T04:46:03.356 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127789.log 2024-04-25T04:46:03.357 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130968.log.gz 2024-04-25T04:46:03.357 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142820.log 2024-04-25T04:46:03.358 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127789.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127789.log.gz 2024-04-25T04:46:03.358 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68508.log 2024-04-25T04:46:03.359 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142820.log.gz 2024-04-25T04:46:03.359 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41736.log 2024-04-25T04:46:03.360 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68508.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68508.log.gz 2024-04-25T04:46:03.360 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105935.log 2024-04-25T04:46:03.361 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41736.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41736.log.gz 2024-04-25T04:46:03.361 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42676.log 2024-04-25T04:46:03.361 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105935.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105935.log.gz 2024-04-25T04:46:03.362 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130088.log 2024-04-25T04:46:03.362 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42676.log.gz 2024-04-25T04:46:03.363 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122928.log 2024-04-25T04:46:03.363 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130088.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130088.log.gz 2024-04-25T04:46:03.364 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110240.log 2024-04-25T04:46:03.364 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122928.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122928.log.gz 2024-04-25T04:46:03.364 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94318.log 2024-04-25T04:46:03.365 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110240.log.gz 2024-04-25T04:46:03.365 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40902.log 2024-04-25T04:46:03.366 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94318.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94318.log.gz 2024-04-25T04:46:03.366 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72871.log 2024-04-25T04:46:03.367 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40902.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40902.log.gz 2024-04-25T04:46:03.367 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84665.log 2024-04-25T04:46:03.368 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72871.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72871.log.gz 2024-04-25T04:46:03.368 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77675.log 2024-04-25T04:46:03.369 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84665.log.gz 2024-04-25T04:46:03.369 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133178.log 2024-04-25T04:46:03.369 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77675.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77675.log.gz 2024-04-25T04:46:03.370 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71352.log 2024-04-25T04:46:03.370 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133178.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133178.log.gz 2024-04-25T04:46:03.371 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127705.log 2024-04-25T04:46:03.371 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71352.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71352.log.gz 2024-04-25T04:46:03.371 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80299.log 2024-04-25T04:46:03.372 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127705.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127705.log.gz 2024-04-25T04:46:03.372 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101074.log 2024-04-25T04:46:03.373 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80299.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80299.log.gz 2024-04-25T04:46:03.373 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159652.log 2024-04-25T04:46:03.374 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101074.log.gz 2024-04-25T04:46:03.374 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61725.log 2024-04-25T04:46:03.375 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159652.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159652.log.gz 2024-04-25T04:46:03.375 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148605.log 2024-04-25T04:46:03.376 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61725.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61725.log.gz 2024-04-25T04:46:03.376 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54901.log 2024-04-25T04:46:03.376 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148605.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148605.log.gz 2024-04-25T04:46:03.377 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98948.log 2024-04-25T04:46:03.377 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54901.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54901.log.gz 2024-04-25T04:46:03.377 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55413.log 2024-04-25T04:46:03.378 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98948.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98948.log.gz 2024-04-25T04:46:03.378 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153166.log 2024-04-25T04:46:03.379 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.55413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55413.log.gz 2024-04-25T04:46:03.379 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134329.log 2024-04-25T04:46:03.380 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153166.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153166.log.gz 2024-04-25T04:46:03.380 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120750.log 2024-04-25T04:46:03.381 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134329.log.gz 2024-04-25T04:46:03.381 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69902.log 2024-04-25T04:46:03.381 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120750.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120750.log.gz 2024-04-25T04:46:03.382 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100656.log 2024-04-25T04:46:03.382 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69902.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69902.log.gz 2024-04-25T04:46:03.383 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62271.log 2024-04-25T04:46:03.383 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100656.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100656.log.gz 2024-04-25T04:46:03.383 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130908.log 2024-04-25T04:46:03.384 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62271.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62271.log.gz 2024-04-25T04:46:03.384 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61474.log 2024-04-25T04:46:03.385 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130908.log.gz 2024-04-25T04:46:03.385 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92082.log 2024-04-25T04:46:03.386 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61474.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61474.log.gz 2024-04-25T04:46:03.386 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144125.log 2024-04-25T04:46:03.387 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92082.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92082.log.gz 2024-04-25T04:46:03.387 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141616.log 2024-04-25T04:46:03.388 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144125.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144125.log.gz 2024-04-25T04:46:03.388 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80007.log 2024-04-25T04:46:03.389 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141616.log.gz 2024-04-25T04:46:03.389 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134537.log 2024-04-25T04:46:03.390 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80007.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80007.log.gz 2024-04-25T04:46:03.390 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97703.log 2024-04-25T04:46:03.390 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.134537.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134537.log.gz 2024-04-25T04:46:03.391 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88554.log 2024-04-25T04:46:03.391 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97703.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97703.log.gz 2024-04-25T04:46:03.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45753.log 2024-04-25T04:46:03.392 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88554.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88554.log.gz 2024-04-25T04:46:03.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147447.log 2024-04-25T04:46:03.393 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45753.log.gz 2024-04-25T04:46:03.393 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112562.log 2024-04-25T04:46:03.394 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147447.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147447.log.gz 2024-04-25T04:46:03.394 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67459.log 2024-04-25T04:46:03.395 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112562.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112562.log.gz 2024-04-25T04:46:03.395 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69033.log 2024-04-25T04:46:03.396 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67459.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67459.log.gz 2024-04-25T04:46:03.396 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151504.log 2024-04-25T04:46:03.396 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69033.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69033.log.gz 2024-04-25T04:46:03.397 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128531.log 2024-04-25T04:46:03.397 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151504.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151504.log.gz 2024-04-25T04:46:03.398 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122333.log 2024-04-25T04:46:03.398 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128531.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128531.log.gz 2024-04-25T04:46:03.399 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59762.log 2024-04-25T04:46:03.399 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122333.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122333.log.gz 2024-04-25T04:46:03.399 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109135.log 2024-04-25T04:46:03.400 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59762.log.gz 2024-04-25T04:46:03.400 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109429.log 2024-04-25T04:46:03.401 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109135.log.gz 2024-04-25T04:46:03.401 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85170.log 2024-04-25T04:46:03.402 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109429.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109429.log.gz 2024-04-25T04:46:03.402 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54467.log 2024-04-25T04:46:03.403 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85170.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85170.log.gz 2024-04-25T04:46:03.403 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86995.log 2024-04-25T04:46:03.404 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54467.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54467.log.gz 2024-04-25T04:46:03.404 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88829.log 2024-04-25T04:46:03.405 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86995.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86995.log.gz 2024-04-25T04:46:03.405 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138909.log 2024-04-25T04:46:03.405 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88829.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88829.log.gz 2024-04-25T04:46:03.406 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73418.log 2024-04-25T04:46:03.407 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138909.log.gz 2024-04-25T04:46:03.407 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132388.log 2024-04-25T04:46:03.407 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73418.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73418.log.gz 2024-04-25T04:46:03.408 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151243.log 2024-04-25T04:46:03.408 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132388.log.gz 2024-04-25T04:46:03.409 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108696.log 2024-04-25T04:46:03.409 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151243.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151243.log.gz 2024-04-25T04:46:03.410 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93830.log 2024-04-25T04:46:03.410 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108696.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108696.log.gz 2024-04-25T04:46:03.411 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119117.log 2024-04-25T04:46:03.411 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93830.log.gz 2024-04-25T04:46:03.411 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43542.log 2024-04-25T04:46:03.412 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119117.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119117.log.gz 2024-04-25T04:46:03.412 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48683.log 2024-04-25T04:46:03.413 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43542.log.gz 2024-04-25T04:46:03.413 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129499.log 2024-04-25T04:46:03.413 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48683.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48683.log.gz 2024-04-25T04:46:03.414 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82049.log 2024-04-25T04:46:03.414 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129499.log.gz 2024-04-25T04:46:03.415 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87191.log 2024-04-25T04:46:03.415 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82049.log.gz 2024-04-25T04:46:03.416 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159626.log 2024-04-25T04:46:03.416 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87191.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87191.log.gz 2024-04-25T04:46:03.417 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80237.log 2024-04-25T04:46:03.417 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159626.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159626.log.gz 2024-04-25T04:46:03.417 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91558.log 2024-04-25T04:46:03.418 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80237.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80237.log.gz 2024-04-25T04:46:03.418 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154978.log 2024-04-25T04:46:03.419 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91558.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91558.log.gz 2024-04-25T04:46:03.419 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124656.log 2024-04-25T04:46:03.420 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154978.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154978.log.gz 2024-04-25T04:46:03.420 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56709.log 2024-04-25T04:46:03.421 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124656.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124656.log.gz 2024-04-25T04:46:03.421 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142694.log 2024-04-25T04:46:03.421 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56709.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56709.log.gz 2024-04-25T04:46:03.422 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138454.log 2024-04-25T04:46:03.422 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142694.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142694.log.gz 2024-04-25T04:46:03.423 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81210.log 2024-04-25T04:46:03.423 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138454.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138454.log.gz 2024-04-25T04:46:03.423 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81064.log 2024-04-25T04:46:03.424 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81210.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81210.log.gz 2024-04-25T04:46:03.424 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113937.log 2024-04-25T04:46:03.425 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81064.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81064.log.gz 2024-04-25T04:46:03.425 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138125.log 2024-04-25T04:46:03.426 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113937.log.gz 2024-04-25T04:46:03.426 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154681.log 2024-04-25T04:46:03.427 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138125.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138125.log.gz 2024-04-25T04:46:03.427 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59048.log 2024-04-25T04:46:03.428 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154681.log.gz 2024-04-25T04:46:03.428 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65149.log 2024-04-25T04:46:03.429 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59048.log.gz 2024-04-25T04:46:03.429 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124469.log 2024-04-25T04:46:03.430 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65149.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65149.log.gz 2024-04-25T04:46:03.430 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48491.log 2024-04-25T04:46:03.430 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124469.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124469.log.gz 2024-04-25T04:46:03.430 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56750.log 2024-04-25T04:46:03.431 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48491.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48491.log.gz 2024-04-25T04:46:03.431 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95049.log 2024-04-25T04:46:03.432 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56750.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56750.log.gz 2024-04-25T04:46:03.432 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118281.log 2024-04-25T04:46:03.433 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95049.log.gz 2024-04-25T04:46:03.433 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104079.log 2024-04-25T04:46:03.434 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118281.log.gz 2024-04-25T04:46:03.434 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153666.log 2024-04-25T04:46:03.435 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104079.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104079.log.gz 2024-04-25T04:46:03.435 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154934.log 2024-04-25T04:46:03.435 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153666.log.gz 2024-04-25T04:46:03.436 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56876.log 2024-04-25T04:46:03.436 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154934.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154934.log.gz 2024-04-25T04:46:03.437 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155480.log 2024-04-25T04:46:03.437 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56876.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56876.log.gz 2024-04-25T04:46:03.437 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100998.log 2024-04-25T04:46:03.438 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155480.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155480.log.gz 2024-04-25T04:46:03.438 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72704.log 2024-04-25T04:46:03.439 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100998.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100998.log.gz 2024-04-25T04:46:03.439 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57022.log 2024-04-25T04:46:03.440 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72704.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72704.log.gz 2024-04-25T04:46:03.440 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156616.log 2024-04-25T04:46:03.440 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57022.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57022.log.gz 2024-04-25T04:46:03.441 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76622.log 2024-04-25T04:46:03.441 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156616.log.gz 2024-04-25T04:46:03.442 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142602.log 2024-04-25T04:46:03.442 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76622.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76622.log.gz 2024-04-25T04:46:03.442 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92720.log 2024-04-25T04:46:03.443 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142602.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142602.log.gz 2024-04-25T04:46:03.443 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157252.log 2024-04-25T04:46:03.444 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92720.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92720.log.gz 2024-04-25T04:46:03.444 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93789.log 2024-04-25T04:46:03.445 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157252.log.gz 2024-04-25T04:46:03.445 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127118.log 2024-04-25T04:46:03.446 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93789.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93789.log.gz 2024-04-25T04:46:03.446 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120684.log 2024-04-25T04:46:03.446 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127118.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127118.log.gz 2024-04-25T04:46:03.447 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85764.log 2024-04-25T04:46:03.447 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120684.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120684.log.gz 2024-04-25T04:46:03.447 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103930.log 2024-04-25T04:46:03.448 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85764.log.gz 2024-04-25T04:46:03.448 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56811.log 2024-04-25T04:46:03.449 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103930.log.gz 2024-04-25T04:46:03.449 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87547.log 2024-04-25T04:46:03.450 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56811.log.gz 2024-04-25T04:46:03.450 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60189.log 2024-04-25T04:46:03.451 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87547.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87547.log.gz 2024-04-25T04:46:03.451 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124308.log 2024-04-25T04:46:03.452 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60189.log.gz 2024-04-25T04:46:03.452 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84666.log 2024-04-25T04:46:03.452 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124308.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124308.log.gz 2024-04-25T04:46:03.453 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128594.log 2024-04-25T04:46:03.453 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84666.log.gz 2024-04-25T04:46:03.453 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113160.log 2024-04-25T04:46:03.454 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128594.log.gz 2024-04-25T04:46:03.454 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56458.log 2024-04-25T04:46:03.455 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113160.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113160.log.gz 2024-04-25T04:46:03.455 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116703.log 2024-04-25T04:46:03.456 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56458.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56458.log.gz 2024-04-25T04:46:03.456 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129478.log 2024-04-25T04:46:03.457 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116703.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116703.log.gz 2024-04-25T04:46:03.457 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113502.log 2024-04-25T04:46:03.458 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129478.log.gz 2024-04-25T04:46:03.458 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64096.log 2024-04-25T04:46:03.458 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113502.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113502.log.gz 2024-04-25T04:46:03.458 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120390.log 2024-04-25T04:46:03.459 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64096.log.gz 2024-04-25T04:46:03.459 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101877.log 2024-04-25T04:46:03.460 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120390.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120390.log.gz 2024-04-25T04:46:03.460 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161509.log 2024-04-25T04:46:03.461 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101877.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101877.log.gz 2024-04-25T04:46:03.461 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41753.log 2024-04-25T04:46:03.462 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161509.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161509.log.gz 2024-04-25T04:46:03.462 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43787.log 2024-04-25T04:46:03.462 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41753.log.gz 2024-04-25T04:46:03.463 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118886.log 2024-04-25T04:46:03.463 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43787.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43787.log.gz 2024-04-25T04:46:03.463 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72198.log 2024-04-25T04:46:03.464 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118886.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118886.log.gz 2024-04-25T04:46:03.464 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95009.log 2024-04-25T04:46:03.465 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72198.log.gz 2024-04-25T04:46:03.465 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61366.log 2024-04-25T04:46:03.466 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95009.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95009.log.gz 2024-04-25T04:46:03.466 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158108.log 2024-04-25T04:46:03.467 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61366.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61366.log.gz 2024-04-25T04:46:03.467 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141884.log 2024-04-25T04:46:03.468 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158108.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158108.log.gz 2024-04-25T04:46:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135965.log 2024-04-25T04:46:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141884.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141884.log.gz 2024-04-25T04:46:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160613.log 2024-04-25T04:46:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135965.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135965.log.gz 2024-04-25T04:46:03.470 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152574.log 2024-04-25T04:46:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.160613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160613.log.gz 2024-04-25T04:46:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122047.log 2024-04-25T04:46:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152574.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152574.log.gz 2024-04-25T04:46:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131967.log 2024-04-25T04:46:03.472 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122047.log.gz 2024-04-25T04:46:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125833.log 2024-04-25T04:46:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131967.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131967.log.gz 2024-04-25T04:46:03.473 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160035.log 2024-04-25T04:46:03.474 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125833.log.gz 2024-04-25T04:46:03.474 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104582.log 2024-04-25T04:46:03.475 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.160035.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160035.log.gz 2024-04-25T04:46:03.475 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109514.log 2024-04-25T04:46:03.476 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104582.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104582.log.gz 2024-04-25T04:46:03.476 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118344.log 2024-04-25T04:46:03.477 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109514.log.gz 2024-04-25T04:46:03.477 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148833.log 2024-04-25T04:46:03.478 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118344.log.gz 2024-04-25T04:46:03.478 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137144.log 2024-04-25T04:46:03.478 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148833.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148833.log.gz 2024-04-25T04:46:03.479 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104633.log 2024-04-25T04:46:03.479 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137144.log.gz 2024-04-25T04:46:03.479 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107831.log 2024-04-25T04:46:03.480 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104633.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104633.log.gz 2024-04-25T04:46:03.481 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142431.log 2024-04-25T04:46:03.481 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107831.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107831.log.gz 2024-04-25T04:46:03.481 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62785.log 2024-04-25T04:46:03.482 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142431.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142431.log.gz 2024-04-25T04:46:03.482 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152993.log 2024-04-25T04:46:03.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62785.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62785.log.gz 2024-04-25T04:46:03.483 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51167.log 2024-04-25T04:46:03.484 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152993.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152993.log.gz 2024-04-25T04:46:03.484 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136880.log 2024-04-25T04:46:03.485 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51167.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51167.log.gz 2024-04-25T04:46:03.485 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129961.log 2024-04-25T04:46:03.485 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136880.log.gz 2024-04-25T04:46:03.486 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130216.log 2024-04-25T04:46:03.486 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129961.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129961.log.gz 2024-04-25T04:46:03.486 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101462.log 2024-04-25T04:46:03.487 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130216.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130216.log.gz 2024-04-25T04:46:03.487 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66878.log 2024-04-25T04:46:03.488 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101462.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101462.log.gz 2024-04-25T04:46:03.488 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67586.log 2024-04-25T04:46:03.489 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66878.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66878.log.gz 2024-04-25T04:46:03.489 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51390.log 2024-04-25T04:46:03.490 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.67586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67586.log.gz 2024-04-25T04:46:03.490 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108254.log 2024-04-25T04:46:03.490 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51390.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51390.log.gz 2024-04-25T04:46:03.491 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81943.log 2024-04-25T04:46:03.491 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108254.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108254.log.gz 2024-04-25T04:46:03.491 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149116.log 2024-04-25T04:46:03.492 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81943.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81943.log.gz 2024-04-25T04:46:03.492 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129288.log 2024-04-25T04:46:03.493 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149116.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149116.log.gz 2024-04-25T04:46:03.493 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88749.log 2024-04-25T04:46:03.494 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129288.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129288.log.gz 2024-04-25T04:46:03.494 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122442.log 2024-04-25T04:46:03.495 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88749.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88749.log.gz 2024-04-25T04:46:03.495 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149578.log 2024-04-25T04:46:03.496 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122442.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122442.log.gz 2024-04-25T04:46:03.496 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53292.log 2024-04-25T04:46:03.496 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149578.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149578.log.gz 2024-04-25T04:46:03.496 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59088.log 2024-04-25T04:46:03.497 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53292.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53292.log.gz 2024-04-25T04:46:03.497 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109342.log 2024-04-25T04:46:03.498 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59088.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59088.log.gz 2024-04-25T04:46:03.498 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55830.log 2024-04-25T04:46:03.499 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109342.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109342.log.gz 2024-04-25T04:46:03.499 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148673.log 2024-04-25T04:46:03.499 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55830.log.gz 2024-04-25T04:46:03.500 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81586.log 2024-04-25T04:46:03.500 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148673.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148673.log.gz 2024-04-25T04:46:03.501 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45545.log 2024-04-25T04:46:03.502 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81586.log.gz 2024-04-25T04:46:03.502 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140112.log 2024-04-25T04:46:03.502 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45545.log.gz 2024-04-25T04:46:03.502 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121386.log 2024-04-25T04:46:03.503 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140112.log.gz 2024-04-25T04:46:03.503 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86824.log 2024-04-25T04:46:03.504 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121386.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121386.log.gz 2024-04-25T04:46:03.504 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157683.log 2024-04-25T04:46:03.505 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86824.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86824.log.gz 2024-04-25T04:46:03.505 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123519.log 2024-04-25T04:46:03.506 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157683.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157683.log.gz 2024-04-25T04:46:03.506 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136699.log 2024-04-25T04:46:03.507 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123519.log.gz 2024-04-25T04:46:03.507 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80510.log 2024-04-25T04:46:03.508 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136699.log.gz 2024-04-25T04:46:03.508 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160771.log 2024-04-25T04:46:03.508 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80510.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80510.log.gz 2024-04-25T04:46:03.509 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71781.log 2024-04-25T04:46:03.509 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.160771.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160771.log.gz 2024-04-25T04:46:03.509 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97329.log 2024-04-25T04:46:03.510 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71781.log.gz 2024-04-25T04:46:03.510 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116548.log 2024-04-25T04:46:03.511 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97329.log.gz 2024-04-25T04:46:03.511 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47843.log 2024-04-25T04:46:03.512 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116548.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116548.log.gz 2024-04-25T04:46:03.512 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146016.log 2024-04-25T04:46:03.513 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47843.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47843.log.gz 2024-04-25T04:46:03.513 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66842.log 2024-04-25T04:46:03.513 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146016.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146016.log.gz 2024-04-25T04:46:03.514 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146537.log 2024-04-25T04:46:03.515 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66842.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66842.log.gz 2024-04-25T04:46:03.515 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98586.log 2024-04-25T04:46:03.515 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146537.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146537.log.gz 2024-04-25T04:46:03.516 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41011.log 2024-04-25T04:46:03.516 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98586.log.gz 2024-04-25T04:46:03.516 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65133.log 2024-04-25T04:46:03.517 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41011.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41011.log.gz 2024-04-25T04:46:03.517 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56665.log 2024-04-25T04:46:03.518 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65133.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65133.log.gz 2024-04-25T04:46:03.518 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91665.log 2024-04-25T04:46:03.519 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56665.log.gz 2024-04-25T04:46:03.519 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152072.log 2024-04-25T04:46:03.520 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91665.log.gz 2024-04-25T04:46:03.520 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111140.log 2024-04-25T04:46:03.520 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152072.log.gz 2024-04-25T04:46:03.521 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129289.log 2024-04-25T04:46:03.521 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111140.log.gz 2024-04-25T04:46:03.522 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61092.log 2024-04-25T04:46:03.522 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129289.log.gz 2024-04-25T04:46:03.523 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63505.log 2024-04-25T04:46:03.523 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61092.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61092.log.gz 2024-04-25T04:46:03.524 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104798.log 2024-04-25T04:46:03.524 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63505.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63505.log.gz 2024-04-25T04:46:03.524 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142204.log 2024-04-25T04:46:03.525 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104798.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104798.log.gz 2024-04-25T04:46:03.525 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127957.log 2024-04-25T04:46:03.526 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142204.log.gz 2024-04-25T04:46:03.526 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96132.log 2024-04-25T04:46:03.527 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127957.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127957.log.gz 2024-04-25T04:46:03.527 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97561.log 2024-04-25T04:46:03.528 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96132.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96132.log.gz 2024-04-25T04:46:03.528 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73587.log 2024-04-25T04:46:03.529 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97561.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97561.log.gz 2024-04-25T04:46:03.529 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65160.log 2024-04-25T04:46:03.529 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73587.log.gz 2024-04-25T04:46:03.530 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154252.log 2024-04-25T04:46:03.530 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65160.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65160.log.gz 2024-04-25T04:46:03.530 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158000.log 2024-04-25T04:46:03.531 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154252.log.gz 2024-04-25T04:46:03.531 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127037.log 2024-04-25T04:46:03.532 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158000.log.gz 2024-04-25T04:46:03.532 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144522.log 2024-04-25T04:46:03.533 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127037.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127037.log.gz 2024-04-25T04:46:03.533 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80111.log 2024-04-25T04:46:03.534 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144522.log.gz 2024-04-25T04:46:03.534 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153310.log 2024-04-25T04:46:03.535 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80111.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80111.log.gz 2024-04-25T04:46:03.535 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97117.log 2024-04-25T04:46:03.535 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153310.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153310.log.gz 2024-04-25T04:46:03.536 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99775.log 2024-04-25T04:46:03.536 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97117.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97117.log.gz 2024-04-25T04:46:03.536 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98670.log 2024-04-25T04:46:03.537 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99775.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99775.log.gz 2024-04-25T04:46:03.537 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81360.log 2024-04-25T04:46:03.538 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98670.log.gz 2024-04-25T04:46:03.538 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53783.log 2024-04-25T04:46:03.539 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81360.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81360.log.gz 2024-04-25T04:46:03.539 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93162.log 2024-04-25T04:46:03.539 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.53783.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53783.log.gz 2024-04-25T04:46:03.540 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65738.log 2024-04-25T04:46:03.540 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93162.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93162.log.gz 2024-04-25T04:46:03.541 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111137.log 2024-04-25T04:46:03.541 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.65738.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65738.log.gz 2024-04-25T04:46:03.542 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94911.log 2024-04-25T04:46:03.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.111137.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111137.log.gz 2024-04-25T04:46:03.542 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62961.log 2024-04-25T04:46:03.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94911.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94911.log.gz 2024-04-25T04:46:03.543 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60855.log 2024-04-25T04:46:03.544 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62961.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62961.log.gz 2024-04-25T04:46:03.544 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142925.log 2024-04-25T04:46:03.545 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60855.log.gz 2024-04-25T04:46:03.545 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84579.log 2024-04-25T04:46:03.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142925.log.gz 2024-04-25T04:46:03.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59616.log 2024-04-25T04:46:03.547 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84579.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84579.log.gz 2024-04-25T04:46:03.547 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75996.log 2024-04-25T04:46:03.548 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59616.log.gz 2024-04-25T04:46:03.548 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131080.log 2024-04-25T04:46:03.548 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.75996.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75996.log.gz 2024-04-25T04:46:03.549 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53277.log 2024-04-25T04:46:03.549 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131080.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131080.log.gz 2024-04-25T04:46:03.550 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93874.log 2024-04-25T04:46:03.550 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53277.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53277.log.gz 2024-04-25T04:46:03.550 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157617.log 2024-04-25T04:46:03.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93874.log.gz 2024-04-25T04:46:03.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140875.log 2024-04-25T04:46:03.552 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157617.log.gz 2024-04-25T04:46:03.552 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45587.log 2024-04-25T04:46:03.553 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140875.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140875.log.gz 2024-04-25T04:46:03.553 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98958.log 2024-04-25T04:46:03.554 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45587.log.gz 2024-04-25T04:46:03.554 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153894.log 2024-04-25T04:46:03.555 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98958.log.gz 2024-04-25T04:46:03.555 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137920.log 2024-04-25T04:46:03.555 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153894.log.gz 2024-04-25T04:46:03.555 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87893.log 2024-04-25T04:46:03.556 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137920.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137920.log.gz 2024-04-25T04:46:03.556 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65408.log 2024-04-25T04:46:03.557 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87893.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87893.log.gz 2024-04-25T04:46:03.557 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62089.log 2024-04-25T04:46:03.558 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65408.log.gz 2024-04-25T04:46:03.558 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110551.log 2024-04-25T04:46:03.559 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62089.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62089.log.gz 2024-04-25T04:46:03.559 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73165.log 2024-04-25T04:46:03.560 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110551.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110551.log.gz 2024-04-25T04:46:03.560 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88328.log 2024-04-25T04:46:03.561 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73165.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73165.log.gz 2024-04-25T04:46:03.561 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85960.log 2024-04-25T04:46:03.561 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88328.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88328.log.gz 2024-04-25T04:46:03.562 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129140.log 2024-04-25T04:46:03.562 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85960.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85960.log.gz 2024-04-25T04:46:03.562 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133847.log 2024-04-25T04:46:03.563 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129140.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129140.log.gz 2024-04-25T04:46:03.563 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64218.log 2024-04-25T04:46:03.564 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133847.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133847.log.gz 2024-04-25T04:46:03.564 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157998.log 2024-04-25T04:46:03.565 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64218.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64218.log.gz 2024-04-25T04:46:03.565 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44041.log 2024-04-25T04:46:03.566 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157998.log.gz 2024-04-25T04:46:03.566 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114894.log 2024-04-25T04:46:03.567 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44041.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44041.log.gz 2024-04-25T04:46:03.567 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102939.log 2024-04-25T04:46:03.567 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114894.log.gz 2024-04-25T04:46:03.568 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108699.log 2024-04-25T04:46:03.568 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102939.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102939.log.gz 2024-04-25T04:46:03.568 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120489.log 2024-04-25T04:46:03.569 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108699.log.gz 2024-04-25T04:46:03.569 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65461.log 2024-04-25T04:46:03.570 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120489.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120489.log.gz 2024-04-25T04:46:03.570 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120490.log 2024-04-25T04:46:03.571 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65461.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65461.log.gz 2024-04-25T04:46:03.571 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127728.log 2024-04-25T04:46:03.571 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120490.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120490.log.gz 2024-04-25T04:46:03.572 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150719.log 2024-04-25T04:46:03.572 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127728.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127728.log.gz 2024-04-25T04:46:03.573 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136375.log 2024-04-25T04:46:03.573 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150719.log.gz 2024-04-25T04:46:03.573 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82470.log 2024-04-25T04:46:03.574 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136375.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136375.log.gz 2024-04-25T04:46:03.574 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154892.log 2024-04-25T04:46:03.575 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82470.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82470.log.gz 2024-04-25T04:46:03.575 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63295.log 2024-04-25T04:46:03.576 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154892.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154892.log.gz 2024-04-25T04:46:03.576 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104139.log 2024-04-25T04:46:03.577 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63295.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63295.log.gz 2024-04-25T04:46:03.577 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97733.log 2024-04-25T04:46:03.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104139.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104139.log.gz 2024-04-25T04:46:03.578 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136281.log 2024-04-25T04:46:03.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97733.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97733.log.gz 2024-04-25T04:46:03.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78703.log 2024-04-25T04:46:03.579 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136281.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136281.log.gz 2024-04-25T04:46:03.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55061.log 2024-04-25T04:46:03.580 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78703.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78703.log.gz 2024-04-25T04:46:03.580 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95375.log 2024-04-25T04:46:03.581 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55061.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55061.log.gz 2024-04-25T04:46:03.581 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150895.log 2024-04-25T04:46:03.582 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95375.log.gz 2024-04-25T04:46:03.582 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79629.log 2024-04-25T04:46:03.583 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150895.log.gz 2024-04-25T04:46:03.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154515.log 2024-04-25T04:46:03.584 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79629.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79629.log.gz 2024-04-25T04:46:03.584 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51626.log 2024-04-25T04:46:03.584 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154515.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154515.log.gz 2024-04-25T04:46:03.585 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74954.log 2024-04-25T04:46:03.585 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51626.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51626.log.gz 2024-04-25T04:46:03.586 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157351.log 2024-04-25T04:46:03.586 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74954.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74954.log.gz 2024-04-25T04:46:03.586 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134544.log 2024-04-25T04:46:03.587 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157351.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157351.log.gz 2024-04-25T04:46:03.587 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132849.log 2024-04-25T04:46:03.588 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.134544.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134544.log.gz 2024-04-25T04:46:03.588 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153035.log 2024-04-25T04:46:03.589 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132849.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132849.log.gz 2024-04-25T04:46:03.589 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137108.log 2024-04-25T04:46:03.589 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153035.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153035.log.gz 2024-04-25T04:46:03.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55996.log 2024-04-25T04:46:03.590 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137108.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137108.log.gz 2024-04-25T04:46:03.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59338.log 2024-04-25T04:46:03.591 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55996.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55996.log.gz 2024-04-25T04:46:03.591 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136839.log 2024-04-25T04:46:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59338.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59338.log.gz 2024-04-25T04:46:03.592 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120580.log 2024-04-25T04:46:03.593 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136839.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136839.log.gz 2024-04-25T04:46:03.593 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158214.log 2024-04-25T04:46:03.594 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120580.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120580.log.gz 2024-04-25T04:46:03.594 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69169.log 2024-04-25T04:46:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158214.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158214.log.gz 2024-04-25T04:46:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47273.log 2024-04-25T04:46:03.595 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69169.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69169.log.gz 2024-04-25T04:46:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159523.log 2024-04-25T04:46:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47273.log.gz 2024-04-25T04:46:03.596 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137879.log 2024-04-25T04:46:03.597 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159523.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159523.log.gz 2024-04-25T04:46:03.597 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127431.log 2024-04-25T04:46:03.598 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137879.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137879.log.gz 2024-04-25T04:46:03.598 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59130.log 2024-04-25T04:46:03.599 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127431.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127431.log.gz 2024-04-25T04:46:03.599 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90941.log 2024-04-25T04:46:03.600 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59130.log.gz 2024-04-25T04:46:03.600 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97264.log 2024-04-25T04:46:03.600 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90941.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90941.log.gz 2024-04-25T04:46:03.601 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108502.log 2024-04-25T04:46:03.601 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97264.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97264.log.gz 2024-04-25T04:46:03.601 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105429.log 2024-04-25T04:46:03.602 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108502.log.gz 2024-04-25T04:46:03.602 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86489.log 2024-04-25T04:46:03.603 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105429.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105429.log.gz 2024-04-25T04:46:03.603 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105559.log 2024-04-25T04:46:03.604 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86489.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86489.log.gz 2024-04-25T04:46:03.604 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124147.log 2024-04-25T04:46:03.605 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105559.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105559.log.gz 2024-04-25T04:46:03.605 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142516.log 2024-04-25T04:46:03.605 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124147.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124147.log.gz 2024-04-25T04:46:03.606 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142162.log 2024-04-25T04:46:03.606 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142516.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142516.log.gz 2024-04-25T04:46:03.606 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71604.log 2024-04-25T04:46:03.607 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142162.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142162.log.gz 2024-04-25T04:46:03.607 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83844.log 2024-04-25T04:46:03.608 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71604.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71604.log.gz 2024-04-25T04:46:03.608 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44945.log 2024-04-25T04:46:03.609 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83844.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83844.log.gz 2024-04-25T04:46:03.609 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48411.log 2024-04-25T04:46:03.609 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44945.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44945.log.gz 2024-04-25T04:46:03.610 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102107.log 2024-04-25T04:46:03.610 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48411.log.gz 2024-04-25T04:46:03.611 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131920.log 2024-04-25T04:46:03.611 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102107.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102107.log.gz 2024-04-25T04:46:03.611 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49040.log 2024-04-25T04:46:03.612 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131920.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131920.log.gz 2024-04-25T04:46:03.612 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86268.log 2024-04-25T04:46:03.613 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49040.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49040.log.gz 2024-04-25T04:46:03.613 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130090.log 2024-04-25T04:46:03.614 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86268.log.gz 2024-04-25T04:46:03.614 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73993.log 2024-04-25T04:46:03.615 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130090.log.gz 2024-04-25T04:46:03.615 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66214.log 2024-04-25T04:46:03.616 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.73993.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73993.log.gz 2024-04-25T04:46:03.616 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138973.log 2024-04-25T04:46:03.616 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66214.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66214.log.gz 2024-04-25T04:46:03.617 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104632.log 2024-04-25T04:46:03.617 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138973.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138973.log.gz 2024-04-25T04:46:03.617 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67101.log 2024-04-25T04:46:03.618 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104632.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104632.log.gz 2024-04-25T04:46:03.618 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159880.log 2024-04-25T04:46:03.619 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.67101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67101.log.gz 2024-04-25T04:46:03.619 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81001.log 2024-04-25T04:46:03.620 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159880.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159880.log.gz 2024-04-25T04:46:03.620 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51458.log 2024-04-25T04:46:03.621 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81001.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81001.log.gz 2024-04-25T04:46:03.621 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58208.log 2024-04-25T04:46:03.622 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51458.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51458.log.gz 2024-04-25T04:46:03.622 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-04-25T04:46:03.622 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58208.log.gz 2024-04-25T04:46:03.623 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76575.log 2024-04-25T04:46:03.623 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77304.log 2024-04-25T04:46:03.624 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76575.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76575.log.gz 2024-04-25T04:46:03.624 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60424.log 2024-04-25T04:46:03.625 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77304.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77304.log.gz 2024-04-25T04:46:03.626 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42480.log 2024-04-25T04:46:03.626 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60424.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60424.log.gz 2024-04-25T04:46:03.626 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115394.log 2024-04-25T04:46:03.627 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42480.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42480.log.gz 2024-04-25T04:46:03.627 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89448.log 2024-04-25T04:46:03.628 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115394.log.gzgzip 2024-04-25T04:46:03.628 INFO:teuthology.orchestra.run.smithi188.stderr:-5 --verbose -- /var/log/ceph/cluster2-client.admin.76850.log 2024-04-25T04:46:03.629 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89448.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89448.log.gz 2024-04-25T04:46:03.629 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106561.log 2024-04-25T04:46:03.630 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76850.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76850.log.gz 2024-04-25T04:46:03.630 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139327.log 2024-04-25T04:46:03.631 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106561.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106561.log.gz 2024-04-25T04:46:03.631 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78919.log 2024-04-25T04:46:03.632 INFO:teuthology.orchestra.run.smithi188.stderr: 87.5% -- replaced with /var/log/ceph/cluster2.log.gz 2024-04-25T04:46:03.632 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139327.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139327.log.gz 2024-04-25T04:46:03.632 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120832.log 2024-04-25T04:46:03.633 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78919.log.gz 2024-04-25T04:46:03.633 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94382.log 2024-04-25T04:46:03.634 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120832.log.gz 2024-04-25T04:46:03.634 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62350.log 2024-04-25T04:46:03.635 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94382.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94382.log.gz 2024-04-25T04:46:03.635 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155711.log 2024-04-25T04:46:03.636 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62350.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62350.log.gz 2024-04-25T04:46:03.636 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126348.log 2024-04-25T04:46:03.637 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155711.log.gz 2024-04-25T04:46:03.637 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94863.log 2024-04-25T04:46:03.637 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126348.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126348.log.gz 2024-04-25T04:46:03.638 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143662.log 2024-04-25T04:46:03.638 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94863.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94863.log.gz 2024-04-25T04:46:03.639 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48663.log 2024-04-25T04:46:03.639 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143662.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143662.log.gz 2024-04-25T04:46:03.640 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135945.log 2024-04-25T04:46:03.640 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48663.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48663.log.gz 2024-04-25T04:46:03.640 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149314.log 2024-04-25T04:46:03.641 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135945.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135945.log.gz 2024-04-25T04:46:03.641 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119860.log 2024-04-25T04:46:03.642 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149314.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149314.log.gz 2024-04-25T04:46:03.642 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112650.log 2024-04-25T04:46:03.643 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119860.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119860.log.gz 2024-04-25T04:46:03.643 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87124.log 2024-04-25T04:46:03.643 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112650.log.gz 2024-04-25T04:46:03.644 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111596.log 2024-04-25T04:46:03.644 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87124.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87124.log.gz 2024-04-25T04:46:03.645 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89273.log 2024-04-25T04:46:03.646 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111596.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111596.log.gz 2024-04-25T04:46:03.646 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145433.log 2024-04-25T04:46:03.646 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89273.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89273.log.gz 2024-04-25T04:46:03.647 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142926.log 2024-04-25T04:46:03.648 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145433.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145433.log.gz 2024-04-25T04:46:03.648 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60084.log 2024-04-25T04:46:03.648 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142926.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142926.log.gz 2024-04-25T04:46:03.649 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158769.log 2024-04-25T04:46:03.649 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60084.log.gz 2024-04-25T04:46:03.650 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101839.log 2024-04-25T04:46:03.650 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.158769.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158769.log.gz 2024-04-25T04:46:03.651 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101483.log 2024-04-25T04:46:03.651 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101839.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101839.log.gz 2024-04-25T04:46:03.651 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148910.log 2024-04-25T04:46:03.652 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101483.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101483.log.gz 2024-04-25T04:46:03.652 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57354.log 2024-04-25T04:46:03.653 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148910.log.gz 2024-04-25T04:46:03.653 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152911.log 2024-04-25T04:46:03.654 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57354.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57354.log.gz 2024-04-25T04:46:03.654 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101027.log 2024-04-25T04:46:03.655 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152911.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152911.log.gz 2024-04-25T04:46:03.655 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96780.log 2024-04-25T04:46:03.655 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101027.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101027.log.gz 2024-04-25T04:46:03.656 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154849.log 2024-04-25T04:46:03.656 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96780.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96780.log.gz 2024-04-25T04:46:03.657 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132387.log 2024-04-25T04:46:03.657 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154849.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154849.log.gz 2024-04-25T04:46:03.658 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97604.log 2024-04-25T04:46:03.658 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132387.log.gz 2024-04-25T04:46:03.658 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117568.log 2024-04-25T04:46:03.659 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97604.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97604.log.gz 2024-04-25T04:46:03.659 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112737.log 2024-04-25T04:46:03.660 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117568.log.gz 2024-04-25T04:46:03.660 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104454.log 2024-04-25T04:46:03.661 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112737.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112737.log.gz 2024-04-25T04:46:03.661 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139895.log 2024-04-25T04:46:03.662 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104454.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104454.log.gz 2024-04-25T04:46:03.662 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109450.log 2024-04-25T04:46:03.663 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139895.log.gz 2024-04-25T04:46:03.663 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114534.log 2024-04-25T04:46:03.663 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109450.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109450.log.gz 2024-04-25T04:46:03.664 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60650.log 2024-04-25T04:46:03.664 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114534.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114534.log.gz 2024-04-25T04:46:03.665 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54523.log 2024-04-25T04:46:03.665 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60650.log.gz 2024-04-25T04:46:03.666 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122552.log 2024-04-25T04:46:03.666 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54523.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54523.log.gz 2024-04-25T04:46:03.667 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57744.log 2024-04-25T04:46:03.667 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122552.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122552.log.gz 2024-04-25T04:46:03.667 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130800.log 2024-04-25T04:46:03.668 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57744.log.gz 2024-04-25T04:46:03.668 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124864.log 2024-04-25T04:46:03.669 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130800.log.gz 2024-04-25T04:46:03.669 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104330.log 2024-04-25T04:46:03.670 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124864.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124864.log.gz 2024-04-25T04:46:03.670 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86555.log 2024-04-25T04:46:03.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104330.log.gz 2024-04-25T04:46:03.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134588.log 2024-04-25T04:46:03.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86555.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86555.log.gz 2024-04-25T04:46:03.672 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130424.log 2024-04-25T04:46:03.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134588.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134588.log.gz 2024-04-25T04:46:03.673 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131545.log 2024-04-25T04:46:03.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130424.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130424.log.gz 2024-04-25T04:46:03.674 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71696.log 2024-04-25T04:46:03.674 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131545.log.gz 2024-04-25T04:46:03.675 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155523.log 2024-04-25T04:46:03.675 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71696.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71696.log.gz 2024-04-25T04:46:03.675 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114096.log 2024-04-25T04:46:03.676 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155523.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155523.log.gz 2024-04-25T04:46:03.678 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135134.log 2024-04-25T04:46:03.678 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114096.log.gz 2024-04-25T04:46:03.678 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70536.log 2024-04-25T04:46:03.678 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135134.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135134.log.gz 2024-04-25T04:46:03.678 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43607.log 2024-04-25T04:46:03.679 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70536.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70536.log.gz 2024-04-25T04:46:03.679 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64008.log 2024-04-25T04:46:03.680 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43607.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43607.log.gz 2024-04-25T04:46:03.680 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124150.log 2024-04-25T04:46:03.680 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64008.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64008.log.gz 2024-04-25T04:46:03.681 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.90098.log 2024-04-25T04:46:03.681 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124150.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124150.log.gz 2024-04-25T04:46:03.682 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158764.log 2024-04-25T04:46:03.682 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.90098.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.90098.log.gz 2024-04-25T04:46:03.683 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72241.log 2024-04-25T04:46:03.683 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.158764.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158764.log.gz 2024-04-25T04:46:03.683 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129387.log 2024-04-25T04:46:03.684 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72241.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72241.log.gz 2024-04-25T04:46:03.684 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48936.log 2024-04-25T04:46:03.685 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129387.log.gz 2024-04-25T04:46:03.685 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117478.log 2024-04-25T04:46:03.686 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48936.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48936.log.gz 2024-04-25T04:46:03.686 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141422.log 2024-04-25T04:46:03.686 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117478.log.gz 2024-04-25T04:46:03.687 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146874.log 2024-04-25T04:46:03.688 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141422.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141422.log.gz 2024-04-25T04:46:03.688 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72031.log 2024-04-25T04:46:03.689 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146874.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146874.log.gz 2024-04-25T04:46:03.689 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48826.log 2024-04-25T04:46:03.689 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72031.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72031.log.gz 2024-04-25T04:46:03.690 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74009.log 2024-04-25T04:46:03.690 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48826.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48826.log.gz 2024-04-25T04:46:03.691 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80741.log 2024-04-25T04:46:03.691 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.74009.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74009.log.gz 2024-04-25T04:46:03.691 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45330.log 2024-04-25T04:46:03.692 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80741.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80741.log.gz 2024-04-25T04:46:03.692 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92502.log 2024-04-25T04:46:03.693 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45330.log.gz 2024-04-25T04:46:03.693 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62347.log 2024-04-25T04:46:03.694 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92502.log.gz 2024-04-25T04:46:03.694 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43164.log 2024-04-25T04:46:03.695 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62347.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62347.log.gz 2024-04-25T04:46:03.695 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80363.log 2024-04-25T04:46:03.696 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.43164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.43164.log.gz 2024-04-25T04:46:03.696 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142734.log 2024-04-25T04:46:03.697 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80363.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80363.log.gz 2024-04-25T04:46:03.697 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75367.log 2024-04-25T04:46:03.698 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142734.log.gz 2024-04-25T04:46:03.698 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73928.log 2024-04-25T04:46:03.699 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75367.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75367.log.gz 2024-04-25T04:46:03.699 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155521.log 2024-04-25T04:46:03.699 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73928.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73928.log.gz 2024-04-25T04:46:03.700 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154089.log 2024-04-25T04:46:03.700 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155521.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155521.log.gz 2024-04-25T04:46:03.700 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87854.log 2024-04-25T04:46:03.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154089.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154089.log.gz 2024-04-25T04:46:03.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94022.log 2024-04-25T04:46:03.702 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.87854.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87854.log.gz 2024-04-25T04:46:03.702 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98449.log 2024-04-25T04:46:03.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94022.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94022.log.gz 2024-04-25T04:46:03.703 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140834.log 2024-04-25T04:46:03.704 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98449.log.gz 2024-04-25T04:46:03.704 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68353.log 2024-04-25T04:46:03.704 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140834.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140834.log.gz 2024-04-25T04:46:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101789.log 2024-04-25T04:46:03.705 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68353.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68353.log.gz 2024-04-25T04:46:03.706 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56626.log 2024-04-25T04:46:03.707 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101789.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101789.log.gz 2024-04-25T04:46:03.707 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92645.log 2024-04-25T04:46:03.707 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56626.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56626.log.gz 2024-04-25T04:46:03.708 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135428.log 2024-04-25T04:46:03.708 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92645.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92645.log.gz 2024-04-25T04:46:03.708 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117314.log 2024-04-25T04:46:03.709 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.135428.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135428.log.gz 2024-04-25T04:46:03.709 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68299.log 2024-04-25T04:46:03.710 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117314.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117314.log.gz 2024-04-25T04:46:03.710 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155777.log 2024-04-25T04:46:03.711 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68299.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68299.log.gz 2024-04-25T04:46:03.711 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115333.log 2024-04-25T04:46:03.711 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155777.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155777.log.gz 2024-04-25T04:46:03.712 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109449.log 2024-04-25T04:46:03.713 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115333.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115333.log.gz 2024-04-25T04:46:03.713 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61011.log 2024-04-25T04:46:03.714 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109449.log.gz 2024-04-25T04:46:03.714 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125161.log 2024-04-25T04:46:03.714 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61011.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61011.log.gz 2024-04-25T04:46:03.715 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45225.log 2024-04-25T04:46:03.715 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125161.log.gz 2024-04-25T04:46:03.715 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132280.log 2024-04-25T04:46:03.716 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45225.log.gz 2024-04-25T04:46:03.716 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75787.log 2024-04-25T04:46:03.717 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132280.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132280.log.gz 2024-04-25T04:46:03.717 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150112.log 2024-04-25T04:46:03.718 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.75787.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75787.log.gz 2024-04-25T04:46:03.718 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40151.log 2024-04-25T04:46:03.719 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150112.log.gz 2024-04-25T04:46:03.719 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136121.log 2024-04-25T04:46:03.719 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40151.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40151.log.gz 2024-04-25T04:46:03.720 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110484.log 2024-04-25T04:46:03.721 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136121.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136121.log.gz 2024-04-25T04:46:03.721 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49041.log 2024-04-25T04:46:03.721 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110484.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110484.log.gz 2024-04-25T04:46:03.722 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129858.log 2024-04-25T04:46:03.722 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49041.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49041.log.gz 2024-04-25T04:46:03.723 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150868.log 2024-04-25T04:46:03.723 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129858.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129858.log.gz 2024-04-25T04:46:03.723 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97812.log 2024-04-25T04:46:03.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150868.log.gz 2024-04-25T04:46:03.724 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154212.log 2024-04-25T04:46:03.725 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97812.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97812.log.gz 2024-04-25T04:46:03.725 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78808.log 2024-04-25T04:46:03.726 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154212.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154212.log.gz 2024-04-25T04:46:03.726 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79404.log 2024-04-25T04:46:03.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.78808.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78808.log.gz 2024-04-25T04:46:03.727 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117963.log 2024-04-25T04:46:03.728 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79404.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79404.log.gz 2024-04-25T04:46:03.728 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44715.log 2024-04-25T04:46:03.728 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117963.log.gz 2024-04-25T04:46:03.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74078.log 2024-04-25T04:46:03.729 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44715.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44715.log.gz 2024-04-25T04:46:03.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135832.log 2024-04-25T04:46:03.730 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.74078.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74078.log.gz 2024-04-25T04:46:03.730 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159968.log 2024-04-25T04:46:03.731 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.135832.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135832.log.gz 2024-04-25T04:46:03.731 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152828.log 2024-04-25T04:46:03.732 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.159968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159968.log.gz 2024-04-25T04:46:03.732 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63661.log 2024-04-25T04:46:03.733 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152828.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152828.log.gz 2024-04-25T04:46:03.733 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92627.log 2024-04-25T04:46:03.733 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63661.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63661.log.gz 2024-04-25T04:46:03.734 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79754.log 2024-04-25T04:46:03.734 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92627.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92627.log.gz 2024-04-25T04:46:03.735 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63756.log 2024-04-25T04:46:03.735 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79754.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79754.log.gz 2024-04-25T04:46:03.736 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.149336.log 2024-04-25T04:46:03.736 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63756.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63756.log.gz 2024-04-25T04:46:03.736 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149936.log 2024-04-25T04:46:03.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.149336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.149336.log.gz 2024-04-25T04:46:03.737 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77654.log 2024-04-25T04:46:03.738 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149936.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149936.log.gz 2024-04-25T04:46:03.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94461.log 2024-04-25T04:46:03.739 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77654.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77654.log.gz 2024-04-25T04:46:03.739 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82113.log 2024-04-25T04:46:03.740 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94461.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94461.log.gz 2024-04-25T04:46:03.740 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118785.log 2024-04-25T04:46:03.741 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82113.log.gz 2024-04-25T04:46:03.741 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93745.log 2024-04-25T04:46:03.742 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118785.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118785.log.gz 2024-04-25T04:46:03.742 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49205.log 2024-04-25T04:46:03.742 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93745.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93745.log.gz 2024-04-25T04:46:03.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103057.log 2024-04-25T04:46:03.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49205.log.gz 2024-04-25T04:46:03.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58816.log 2024-04-25T04:46:03.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103057.log.gz 2024-04-25T04:46:03.744 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132026.log 2024-04-25T04:46:03.745 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58816.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58816.log.gz 2024-04-25T04:46:03.745 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42564.log 2024-04-25T04:46:03.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132026.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132026.log.gz 2024-04-25T04:46:03.746 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112603.log 2024-04-25T04:46:03.747 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.42564.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42564.log.gz 2024-04-25T04:46:03.747 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43997.log 2024-04-25T04:46:03.748 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.112603.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112603.log.gz 2024-04-25T04:46:03.748 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49585.log 2024-04-25T04:46:03.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43997.log.gz 2024-04-25T04:46:03.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103006.log 2024-04-25T04:46:03.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49585.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49585.log.gz 2024-04-25T04:46:03.750 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57705.log 2024-04-25T04:46:03.750 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.103006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103006.log.gz 2024-04-25T04:46:03.751 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113026.log 2024-04-25T04:46:03.751 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57705.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57705.log.gz 2024-04-25T04:46:03.752 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73913.log 2024-04-25T04:46:03.752 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113026.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113026.log.gz 2024-04-25T04:46:03.752 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138456.log 2024-04-25T04:46:03.753 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.73913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73913.log.gz 2024-04-25T04:46:03.753 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74737.log 2024-04-25T04:46:03.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138456.log.gz 2024-04-25T04:46:03.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125255.log 2024-04-25T04:46:03.755 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74737.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74737.log.gz 2024-04-25T04:46:03.755 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118007.log 2024-04-25T04:46:03.756 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125255.log.gz 2024-04-25T04:46:03.756 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124355.log 2024-04-25T04:46:03.757 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118007.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118007.log.gz 2024-04-25T04:46:03.757 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89361.log 2024-04-25T04:46:03.757 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124355.log.gz 2024-04-25T04:46:03.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62984.log 2024-04-25T04:46:03.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89361.log.gz 2024-04-25T04:46:03.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152596.log 2024-04-25T04:46:03.759 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62984.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62984.log.gz 2024-04-25T04:46:03.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57064.log 2024-04-25T04:46:03.760 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152596.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152596.log.gz 2024-04-25T04:46:03.760 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110077.log 2024-04-25T04:46:03.761 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57064.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57064.log.gz 2024-04-25T04:46:03.761 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80616.log 2024-04-25T04:46:03.762 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110077.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110077.log.gz 2024-04-25T04:46:03.762 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133164.log 2024-04-25T04:46:03.763 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80616.log.gz 2024-04-25T04:46:03.763 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146368.log 2024-04-25T04:46:03.763 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133164.log.gz 2024-04-25T04:46:03.764 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61789.log 2024-04-25T04:46:03.764 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146368.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146368.log.gz 2024-04-25T04:46:03.765 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48180.log 2024-04-25T04:46:03.766 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61789.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61789.log.gz 2024-04-25T04:46:03.766 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68197.log 2024-04-25T04:46:03.766 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48180.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48180.log.gz 2024-04-25T04:46:03.766 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137640.log 2024-04-25T04:46:03.767 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68197.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68197.log.gz 2024-04-25T04:46:03.767 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107750.log 2024-04-25T04:46:03.768 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137640.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137640.log.gz 2024-04-25T04:46:03.768 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144796.log 2024-04-25T04:46:03.769 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107750.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107750.log.gz 2024-04-25T04:46:03.769 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127393.log 2024-04-25T04:46:03.770 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144796.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144796.log.gz 2024-04-25T04:46:03.770 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151035.log 2024-04-25T04:46:03.770 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127393.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127393.log.gz 2024-04-25T04:46:03.771 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52259.log 2024-04-25T04:46:03.771 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151035.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151035.log.gz 2024-04-25T04:46:03.771 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47570.log 2024-04-25T04:46:03.772 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52259.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52259.log.gz 2024-04-25T04:46:03.772 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75777.log 2024-04-25T04:46:03.773 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47570.log.gz 2024-04-25T04:46:03.773 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81820.log 2024-04-25T04:46:03.774 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75777.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75777.log.gz 2024-04-25T04:46:03.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154434.log 2024-04-25T04:46:03.775 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81820.log.gz 2024-04-25T04:46:03.775 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76046.log 2024-04-25T04:46:03.776 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154434.log.gz 2024-04-25T04:46:03.776 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102728.log 2024-04-25T04:46:03.776 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76046.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76046.log.gz 2024-04-25T04:46:03.777 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147650.log 2024-04-25T04:46:03.777 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102728.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102728.log.gz 2024-04-25T04:46:03.777 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107022.log 2024-04-25T04:46:03.778 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147650.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147650.log.gz 2024-04-25T04:46:03.779 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114476.log 2024-04-25T04:46:03.779 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107022.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107022.log.gz 2024-04-25T04:46:03.779 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135854.log 2024-04-25T04:46:03.780 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114476.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114476.log.gz 2024-04-25T04:46:03.780 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107021.log 2024-04-25T04:46:03.781 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135854.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135854.log.gz 2024-04-25T04:46:03.781 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44378.log 2024-04-25T04:46:03.782 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107021.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107021.log.gz 2024-04-25T04:46:03.782 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57664.log 2024-04-25T04:46:03.782 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44378.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44378.log.gz 2024-04-25T04:46:03.783 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106311.log 2024-04-25T04:46:03.783 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57664.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57664.log.gz 2024-04-25T04:46:03.784 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60148.log 2024-04-25T04:46:03.784 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106311.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106311.log.gz 2024-04-25T04:46:03.784 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91623.log 2024-04-25T04:46:03.785 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60148.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60148.log.gz 2024-04-25T04:46:03.785 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59047.log 2024-04-25T04:46:03.786 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91623.log.gz 2024-04-25T04:46:03.786 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42900.log 2024-04-25T04:46:03.787 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59047.log.gz 2024-04-25T04:46:03.787 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97813.log 2024-04-25T04:46:03.787 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42900.log.gz 2024-04-25T04:46:03.788 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130798.log 2024-04-25T04:46:03.788 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97813.log.gz 2024-04-25T04:46:03.788 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132550.log 2024-04-25T04:46:03.789 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130798.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130798.log.gz 2024-04-25T04:46:03.790 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59257.log 2024-04-25T04:46:03.790 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132550.log.gz 2024-04-25T04:46:03.790 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94064.log 2024-04-25T04:46:03.791 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59257.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59257.log.gz 2024-04-25T04:46:03.791 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73586.log 2024-04-25T04:46:03.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94064.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94064.log.gz 2024-04-25T04:46:03.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134237.log 2024-04-25T04:46:03.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73586.log.gz 2024-04-25T04:46:03.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132930.log 2024-04-25T04:46:03.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134237.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134237.log.gz 2024-04-25T04:46:03.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44630.log 2024-04-25T04:46:03.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132930.log.gz 2024-04-25T04:46:03.795 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58984.log 2024-04-25T04:46:03.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44630.log.gz 2024-04-25T04:46:03.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77862.log 2024-04-25T04:46:03.796 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58984.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58984.log.gz 2024-04-25T04:46:03.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65671.log 2024-04-25T04:46:03.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77862.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77862.log.gz 2024-04-25T04:46:03.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79049.log 2024-04-25T04:46:03.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65671.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65671.log.gz 2024-04-25T04:46:03.798 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81822.log 2024-04-25T04:46:03.799 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79049.log.gz 2024-04-25T04:46:03.799 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102901.log 2024-04-25T04:46:03.799 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81822.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81822.log.gz 2024-04-25T04:46:03.800 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147576.log 2024-04-25T04:46:03.800 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102901.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102901.log.gz 2024-04-25T04:46:03.801 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47351.log 2024-04-25T04:46:03.801 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.147576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147576.log.gz 2024-04-25T04:46:03.801 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47329.log 2024-04-25T04:46:03.802 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47351.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47351.log.gz 2024-04-25T04:46:03.802 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96423.log 2024-04-25T04:46:03.803 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47329.log.gz 2024-04-25T04:46:03.803 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148760.log 2024-04-25T04:46:03.804 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96423.log.gz 2024-04-25T04:46:03.804 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135200.log 2024-04-25T04:46:03.805 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148760.log.gz 2024-04-25T04:46:03.805 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81229.log 2024-04-25T04:46:03.806 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135200.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135200.log.gz 2024-04-25T04:46:03.806 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44672.log 2024-04-25T04:46:03.807 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81229.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81229.log.gz 2024-04-25T04:46:03.807 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87639.log 2024-04-25T04:46:03.807 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44672.log.gz 2024-04-25T04:46:03.808 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129919.log 2024-04-25T04:46:03.808 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87639.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87639.log.gz 2024-04-25T04:46:03.808 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42967.log 2024-04-25T04:46:03.809 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129919.log.gz 2024-04-25T04:46:03.809 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123989.log 2024-04-25T04:46:03.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42967.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42967.log.gz 2024-04-25T04:46:03.810 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160572.log 2024-04-25T04:46:03.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123989.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123989.log.gz 2024-04-25T04:46:03.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121998.log 2024-04-25T04:46:03.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160572.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160572.log.gz 2024-04-25T04:46:03.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148699.log 2024-04-25T04:46:03.812 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121998.log.gz 2024-04-25T04:46:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160499.log 2024-04-25T04:46:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148699.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148699.log.gz 2024-04-25T04:46:03.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108255.log 2024-04-25T04:46:03.814 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.160499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160499.log.gz 2024-04-25T04:46:03.814 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81670.log 2024-04-25T04:46:03.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108255.log.gz 2024-04-25T04:46:03.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86352.log 2024-04-25T04:46:03.816 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81670.log.gz 2024-04-25T04:46:03.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155882.log 2024-04-25T04:46:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86352.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86352.log.gz 2024-04-25T04:46:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54084.log 2024-04-25T04:46:03.817 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155882.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155882.log.gz 2024-04-25T04:46:03.818 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111708.log 2024-04-25T04:46:03.818 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54084.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54084.log.gz 2024-04-25T04:46:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143135.log 2024-04-25T04:46:03.819 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111708.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111708.log.gz 2024-04-25T04:46:03.820 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121997.log 2024-04-25T04:46:03.820 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143135.log.gz 2024-04-25T04:46:03.821 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141946.log 2024-04-25T04:46:03.821 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121997.log.gz 2024-04-25T04:46:03.822 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82555.log 2024-04-25T04:46:03.822 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141946.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141946.log.gz 2024-04-25T04:46:03.822 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119590.log 2024-04-25T04:46:03.823 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82555.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82555.log.gz 2024-04-25T04:46:03.823 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85923.log 2024-04-25T04:46:03.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119590.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119590.log.gz 2024-04-25T04:46:03.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95072.log 2024-04-25T04:46:03.825 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85923.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85923.log.gz 2024-04-25T04:46:03.825 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94168.log 2024-04-25T04:46:03.826 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95072.log.gz 2024-04-25T04:46:03.826 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139015.log 2024-04-25T04:46:03.826 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94168.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94168.log.gz 2024-04-25T04:46:03.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131966.log 2024-04-25T04:46:03.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139015.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139015.log.gz 2024-04-25T04:46:03.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49206.log 2024-04-25T04:46:03.828 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131966.log.gz 2024-04-25T04:46:03.828 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136859.log 2024-04-25T04:46:03.829 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49206.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49206.log.gz 2024-04-25T04:46:03.829 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122222.log 2024-04-25T04:46:03.830 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136859.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136859.log.gz 2024-04-25T04:46:03.830 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54815.log 2024-04-25T04:46:03.831 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122222.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122222.log.gz 2024-04-25T04:46:03.831 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113388.log 2024-04-25T04:46:03.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54815.log.gz 2024-04-25T04:46:03.832 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124770.log 2024-04-25T04:46:03.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113388.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113388.log.gz 2024-04-25T04:46:03.833 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67461.log 2024-04-25T04:46:03.833 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124770.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124770.log.gz 2024-04-25T04:46:03.833 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95941.log 2024-04-25T04:46:03.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67461.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67461.log.gz 2024-04-25T04:46:03.834 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79842.log 2024-04-25T04:46:03.835 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95941.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95941.log.gz 2024-04-25T04:46:03.835 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114119.log 2024-04-25T04:46:03.836 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79842.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79842.log.gz 2024-04-25T04:46:03.836 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85120.log 2024-04-25T04:46:03.836 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.114119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114119.log.gz 2024-04-25T04:46:03.837 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88685.log 2024-04-25T04:46:03.837 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85120.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85120.log.gz 2024-04-25T04:46:03.838 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79756.log 2024-04-25T04:46:03.839 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88685.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88685.log.gz 2024-04-25T04:46:03.839 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115544.log 2024-04-25T04:46:03.839 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79756.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79756.log.gz 2024-04-25T04:46:03.839 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124301.log 2024-04-25T04:46:03.840 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115544.log.gz 2024-04-25T04:46:03.840 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117732.log 2024-04-25T04:46:03.841 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124301.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124301.log.gz 2024-04-25T04:46:03.841 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88488.log 2024-04-25T04:46:03.842 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117732.log.gz 2024-04-25T04:46:03.842 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144722.log 2024-04-25T04:46:03.843 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88488.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88488.log.gz 2024-04-25T04:46:03.843 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145766.log 2024-04-25T04:46:03.843 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144722.log.gz 2024-04-25T04:46:03.844 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102726.log 2024-04-25T04:46:03.844 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145766.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145766.log.gz 2024-04-25T04:46:03.845 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.102193.log 2024-04-25T04:46:03.845 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102726.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102726.log.gz 2024-04-25T04:46:03.846 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144081.log 2024-04-25T04:46:03.846 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.102193.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.102193.log.gz 2024-04-25T04:46:03.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40409.log 2024-04-25T04:46:03.847 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144081.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144081.log.gz 2024-04-25T04:46:03.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143663.log 2024-04-25T04:46:03.848 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40409.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40409.log.gz 2024-04-25T04:46:03.848 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84623.log 2024-04-25T04:46:03.849 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143663.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143663.log.gz 2024-04-25T04:46:03.849 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103184.log 2024-04-25T04:46:03.849 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84623.log.gz 2024-04-25T04:46:03.850 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83611.log 2024-04-25T04:46:03.850 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.103184.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103184.log.gz 2024-04-25T04:46:03.851 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43405.log 2024-04-25T04:46:03.851 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83611.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83611.log.gz 2024-04-25T04:46:03.852 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77030.log 2024-04-25T04:46:03.852 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43405.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43405.log.gz 2024-04-25T04:46:03.853 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106774.log 2024-04-25T04:46:03.853 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77030.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77030.log.gz 2024-04-25T04:46:03.853 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131656.log 2024-04-25T04:46:03.854 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106774.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106774.log.gz 2024-04-25T04:46:03.854 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136261.log 2024-04-25T04:46:03.855 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131656.log.gz 2024-04-25T04:46:03.855 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100008.log 2024-04-25T04:46:03.856 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136261.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136261.log.gz 2024-04-25T04:46:03.856 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137087.log 2024-04-25T04:46:03.856 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100008.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100008.log.gz 2024-04-25T04:46:03.857 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142054.log 2024-04-25T04:46:03.857 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.137087.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.137087.log.gz 2024-04-25T04:46:03.857 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150974.log 2024-04-25T04:46:03.858 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142054.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142054.log.gz 2024-04-25T04:46:03.858 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78739.log 2024-04-25T04:46:03.859 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150974.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150974.log.gz 2024-04-25T04:46:03.859 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111010.log 2024-04-25T04:46:03.860 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78739.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78739.log.gz 2024-04-25T04:46:03.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104497.log 2024-04-25T04:46:03.861 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111010.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111010.log.gz 2024-04-25T04:46:03.861 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62238.log 2024-04-25T04:46:03.861 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104497.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104497.log.gz 2024-04-25T04:46:03.862 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110963.log 2024-04-25T04:46:03.862 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62238.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62238.log.gz 2024-04-25T04:46:03.863 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103406.log 2024-04-25T04:46:03.863 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.110963.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110963.log.gz 2024-04-25T04:46:03.864 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97552.log 2024-04-25T04:46:03.864 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103406.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103406.log.gz 2024-04-25T04:46:03.864 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143786.log 2024-04-25T04:46:03.865 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97552.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97552.log.gz 2024-04-25T04:46:03.865 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67840.log 2024-04-25T04:46:03.866 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143786.log.gz 2024-04-25T04:46:03.866 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56352.log 2024-04-25T04:46:03.867 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67840.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67840.log.gz 2024-04-25T04:46:03.867 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124496.log 2024-04-25T04:46:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56352.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56352.log.gz 2024-04-25T04:46:03.868 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73119.log 2024-04-25T04:46:03.869 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124496.log.gz 2024-04-25T04:46:03.869 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99105.log 2024-04-25T04:46:03.870 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73119.log.gz 2024-04-25T04:46:03.870 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153602.log 2024-04-25T04:46:03.871 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99105.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99105.log.gz 2024-04-25T04:46:03.871 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128850.log 2024-04-25T04:46:03.872 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139120.log 2024-04-25T04:46:03.872 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153602.log.gz 2024-04-25T04:46:03.872 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128850.log.gz 2024-04-25T04:46:03.873 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47267.log 2024-04-25T04:46:03.873 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139120.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139120.log.gz 2024-04-25T04:46:03.874 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69442.log 2024-04-25T04:46:03.874 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47267.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47267.log.gz 2024-04-25T04:46:03.875 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88082.log 2024-04-25T04:46:03.875 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69442.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69442.log.gz 2024-04-25T04:46:03.875 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154211.log 2024-04-25T04:46:03.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88082.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88082.log.gz 2024-04-25T04:46:03.876 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104988.log 2024-04-25T04:46:03.877 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154211.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154211.log.gz 2024-04-25T04:46:03.877 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141654.log 2024-04-25T04:46:03.878 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104988.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104988.log.gz 2024-04-25T04:46:03.878 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71436.log 2024-04-25T04:46:03.879 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141654.log.gz 2024-04-25T04:46:03.879 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136167.log 2024-04-25T04:46:03.880 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71436.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71436.log.gz 2024-04-25T04:46:03.880 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94652.log 2024-04-25T04:46:03.882 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136167.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136167.log.gz 2024-04-25T04:46:03.882 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49647.log 2024-04-25T04:46:03.882 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94652.log.gz 2024-04-25T04:46:03.882 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67924.log 2024-04-25T04:46:03.882 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49647.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49647.log.gz 2024-04-25T04:46:03.882 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80742.log 2024-04-25T04:46:03.883 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67924.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67924.log.gz 2024-04-25T04:46:03.883 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114718.log 2024-04-25T04:46:03.884 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80742.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80742.log.gz 2024-04-25T04:46:03.884 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78612.log 2024-04-25T04:46:03.884 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114718.log.gz 2024-04-25T04:46:03.885 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48916.log 2024-04-25T04:46:03.885 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78612.log.gz 2024-04-25T04:46:03.886 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82892.log 2024-04-25T04:46:03.886 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48916.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48916.log.gz 2024-04-25T04:46:03.886 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75740.log 2024-04-25T04:46:03.887 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82892.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82892.log.gz 2024-04-25T04:46:03.887 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136395.log 2024-04-25T04:46:03.888 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75740.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75740.log.gz 2024-04-25T04:46:03.888 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82435.log 2024-04-25T04:46:03.889 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136395.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136395.log.gz 2024-04-25T04:46:03.889 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44101.log 2024-04-25T04:46:03.890 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82435.log.gz 2024-04-25T04:46:03.890 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92928.log 2024-04-25T04:46:03.890 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44101.log.gz 2024-04-25T04:46:03.891 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86601.log 2024-04-25T04:46:03.891 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92928.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92928.log.gz 2024-04-25T04:46:03.892 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116465.log 2024-04-25T04:46:03.893 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86601.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86601.log.gz 2024-04-25T04:46:03.893 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130255.log 2024-04-25T04:46:03.894 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116465.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116465.log.gz 2024-04-25T04:46:03.894 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133783.log 2024-04-25T04:46:03.895 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130255.log.gz 2024-04-25T04:46:03.895 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98968.log 2024-04-25T04:46:03.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133783.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133783.log.gz 2024-04-25T04:46:03.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122198.log 2024-04-25T04:46:03.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98968.log.gz 2024-04-25T04:46:03.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84771.log 2024-04-25T04:46:03.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122198.log.gz 2024-04-25T04:46:03.899 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49627.log 2024-04-25T04:46:03.899 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84771.log.gz 2024-04-25T04:46:03.899 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151473.log 2024-04-25T04:46:03.899 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49627.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49627.log.gz 2024-04-25T04:46:03.899 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161313.log 2024-04-25T04:46:03.901 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151473.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151473.log.gz 2024-04-25T04:46:03.902 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88139.log 2024-04-25T04:46:03.902 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161313.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161313.log.gz 2024-04-25T04:46:03.902 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146796.log 2024-04-25T04:46:03.902 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88139.log.gz 2024-04-25T04:46:03.902 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153433.log 2024-04-25T04:46:03.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146796.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146796.log.gz 2024-04-25T04:46:03.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64222.log 2024-04-25T04:46:03.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153433.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153433.log.gz 2024-04-25T04:46:03.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49042.log 2024-04-25T04:46:03.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64222.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64222.log.gz 2024-04-25T04:46:03.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68721.log 2024-04-25T04:46:03.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49042.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49042.log.gz 2024-04-25T04:46:03.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150551.log 2024-04-25T04:46:03.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68721.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68721.log.gz 2024-04-25T04:46:03.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107410.log 2024-04-25T04:46:03.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150551.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150551.log.gz 2024-04-25T04:46:03.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58750.log 2024-04-25T04:46:03.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107410.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107410.log.gz 2024-04-25T04:46:03.909 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67149.log 2024-04-25T04:46:03.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58750.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58750.log.gz 2024-04-25T04:46:03.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148932.log 2024-04-25T04:46:03.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67149.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67149.log.gz 2024-04-25T04:46:03.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132449.log 2024-04-25T04:46:03.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148932.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148932.log.gz 2024-04-25T04:46:03.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124382.log 2024-04-25T04:46:03.913 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132449.log.gz 2024-04-25T04:46:03.913 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72284.log 2024-04-25T04:46:03.913 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124382.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124382.log.gz 2024-04-25T04:46:03.913 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51734.log 2024-04-25T04:46:03.915 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72284.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72284.log.gz 2024-04-25T04:46:03.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40901.log 2024-04-25T04:46:03.916 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51734.log.gz 2024-04-25T04:46:03.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130149.log 2024-04-25T04:46:03.916 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40901.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40901.log.gz 2024-04-25T04:46:03.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69714.log 2024-04-25T04:46:03.916 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130149.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130149.log.gz 2024-04-25T04:46:03.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44848.log 2024-04-25T04:46:03.917 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69714.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69714.log.gz 2024-04-25T04:46:03.917 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105178.log 2024-04-25T04:46:03.917 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44848.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44848.log.gz 2024-04-25T04:46:03.918 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63051.log 2024-04-25T04:46:03.919 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105178.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105178.log.gz 2024-04-25T04:46:03.919 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115651.log 2024-04-25T04:46:03.919 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63051.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63051.log.gz 2024-04-25T04:46:03.920 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100636.log 2024-04-25T04:46:03.920 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115651.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115651.log.gz 2024-04-25T04:46:03.920 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150554.log 2024-04-25T04:46:03.921 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100636.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100636.log.gz 2024-04-25T04:46:03.921 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56247.log 2024-04-25T04:46:03.922 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150554.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150554.log.gz 2024-04-25T04:46:03.922 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42784.log 2024-04-25T04:46:03.923 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56247.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56247.log.gz 2024-04-25T04:46:03.923 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145832.log 2024-04-25T04:46:03.924 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42784.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42784.log.gz 2024-04-25T04:46:03.924 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96592.log 2024-04-25T04:46:03.924 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145832.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145832.log.gz 2024-04-25T04:46:03.924 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78036.log 2024-04-25T04:46:03.925 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96592.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96592.log.gz 2024-04-25T04:46:03.925 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101556.log 2024-04-25T04:46:03.926 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78036.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78036.log.gz 2024-04-25T04:46:03.926 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106398.log 2024-04-25T04:46:03.927 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101556.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101556.log.gz 2024-04-25T04:46:03.927 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151222.log 2024-04-25T04:46:03.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106398.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106398.log.gz 2024-04-25T04:46:03.928 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153542.log 2024-04-25T04:46:03.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151222.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151222.log.gz 2024-04-25T04:46:03.929 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83845.log 2024-04-25T04:46:03.929 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153542.log.gz 2024-04-25T04:46:03.930 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79416.log 2024-04-25T04:46:03.930 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83845.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83845.log.gz 2024-04-25T04:46:03.931 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136208.log 2024-04-25T04:46:03.931 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79416.log.gz 2024-04-25T04:46:03.931 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129449.log 2024-04-25T04:46:03.932 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136208.log.gz 2024-04-25T04:46:03.932 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42698.log 2024-04-25T04:46:03.933 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129449.log.gz 2024-04-25T04:46:03.933 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78499.log 2024-04-25T04:46:03.934 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.42698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42698.log.gz 2024-04-25T04:46:03.934 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131424.log 2024-04-25T04:46:03.934 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.78499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78499.log.gz 2024-04-25T04:46:03.935 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104268.log 2024-04-25T04:46:03.936 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131424.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131424.log.gz 2024-04-25T04:46:03.936 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89903.log 2024-04-25T04:46:03.936 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104268.log.gz 2024-04-25T04:46:03.936 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100884.log 2024-04-25T04:46:03.937 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89903.log.gz 2024-04-25T04:46:03.937 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92039.log 2024-04-25T04:46:03.938 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100884.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100884.log.gz 2024-04-25T04:46:03.938 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74670.log 2024-04-25T04:46:03.939 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138250.log 2024-04-25T04:46:03.939 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92039.log.gz 2024-04-25T04:46:03.940 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74670.log.gz 2024-04-25T04:46:03.940 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96820.log 2024-04-25T04:46:03.941 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.138250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138250.log.gz 2024-04-25T04:46:03.941 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42852.log 2024-04-25T04:46:03.942 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96820.log.gz 2024-04-25T04:46:03.942 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108485.log 2024-04-25T04:46:03.943 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.42852.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.42852.log.gz 2024-04-25T04:46:03.943 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100770.log 2024-04-25T04:46:03.944 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108485.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108485.log.gz 2024-04-25T04:46:03.944 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80508.log 2024-04-25T04:46:03.944 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100770.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100770.log.gz 2024-04-25T04:46:03.945 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82827.log 2024-04-25T04:46:03.945 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80508.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80508.log.gz 2024-04-25T04:46:03.946 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42028.log 2024-04-25T04:46:03.946 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82827.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82827.log.gz 2024-04-25T04:46:03.947 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45372.log 2024-04-25T04:46:03.947 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.42028.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42028.log.gz 2024-04-25T04:46:03.947 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76185.log 2024-04-25T04:46:03.948 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45372.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45372.log.gz 2024-04-25T04:46:03.948 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141297.log 2024-04-25T04:46:03.949 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.76185.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76185.log.gz 2024-04-25T04:46:03.949 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74669.log 2024-04-25T04:46:03.950 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141297.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141297.log.gz 2024-04-25T04:46:03.950 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125416.log 2024-04-25T04:46:03.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74669.log.gz 2024-04-25T04:46:03.951 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54681.log 2024-04-25T04:46:03.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125416.log.gz 2024-04-25T04:46:03.952 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72762.log 2024-04-25T04:46:03.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54681.log.gz 2024-04-25T04:46:03.953 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59936.log 2024-04-25T04:46:03.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72762.log.gz 2024-04-25T04:46:03.953 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83571.log 2024-04-25T04:46:03.954 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59936.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59936.log.gz 2024-04-25T04:46:03.954 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140292.log 2024-04-25T04:46:03.955 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83571.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83571.log.gz 2024-04-25T04:46:03.955 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130466.log 2024-04-25T04:46:03.956 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140292.log.gz 2024-04-25T04:46:03.956 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84476.log 2024-04-25T04:46:03.957 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130466.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130466.log.gz 2024-04-25T04:46:03.957 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85444.log 2024-04-25T04:46:03.957 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84476.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84476.log.gz 2024-04-25T04:46:03.958 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122553.log 2024-04-25T04:46:03.959 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85444.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85444.log.gz 2024-04-25T04:46:03.959 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110485.log 2024-04-25T04:46:03.959 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122553.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122553.log.gz 2024-04-25T04:46:03.960 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123742.log 2024-04-25T04:46:03.960 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110485.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110485.log.gz 2024-04-25T04:46:03.960 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54011.log 2024-04-25T04:46:03.961 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.123742.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123742.log.gz 2024-04-25T04:46:03.961 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64675.log 2024-04-25T04:46:03.962 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54011.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54011.log.gz 2024-04-25T04:46:03.962 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92292.log 2024-04-25T04:46:03.963 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.64675.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64675.log.gz 2024-04-25T04:46:03.963 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116203.log 2024-04-25T04:46:03.964 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92292.log.gz 2024-04-25T04:46:03.964 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81400.log 2024-04-25T04:46:03.964 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116203.log.gz 2024-04-25T04:46:03.965 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84000.log 2024-04-25T04:46:03.965 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81400.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81400.log.gz 2024-04-25T04:46:03.966 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80719.log 2024-04-25T04:46:03.966 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84000.log.gz 2024-04-25T04:46:03.967 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69715.log 2024-04-25T04:46:03.967 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80719.log.gz 2024-04-25T04:46:03.967 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137370.log 2024-04-25T04:46:03.968 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69715.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69715.log.gz 2024-04-25T04:46:03.968 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156132.log 2024-04-25T04:46:03.969 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137370.log.gz 2024-04-25T04:46:03.969 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68092.log 2024-04-25T04:46:03.970 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156132.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156132.log.gz 2024-04-25T04:46:03.970 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115506.log 2024-04-25T04:46:03.971 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68092.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68092.log.gz 2024-04-25T04:46:03.971 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51637.log 2024-04-25T04:46:03.972 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115506.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115506.log.gz 2024-04-25T04:46:03.972 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57955.log 2024-04-25T04:46:03.972 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51637.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51637.log.gz 2024-04-25T04:46:03.973 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148491.log 2024-04-25T04:46:03.973 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57955.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57955.log.gz 2024-04-25T04:46:03.974 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109307.log 2024-04-25T04:46:03.974 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148491.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148491.log.gz 2024-04-25T04:46:03.974 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93182.log 2024-04-25T04:46:03.975 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109307.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109307.log.gz 2024-04-25T04:46:03.975 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100504.log 2024-04-25T04:46:03.976 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93182.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93182.log.gz 2024-04-25T04:46:03.976 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156135.log 2024-04-25T04:46:03.977 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100504.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100504.log.gz 2024-04-25T04:46:03.977 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143411.log 2024-04-25T04:46:03.978 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156135.log.gz 2024-04-25T04:46:03.978 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66440.log 2024-04-25T04:46:03.979 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143411.log.gz 2024-04-25T04:46:03.979 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78767.log 2024-04-25T04:46:03.980 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66440.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66440.log.gz 2024-04-25T04:46:03.980 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146253.log 2024-04-25T04:46:03.980 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78767.log.gz 2024-04-25T04:46:03.981 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121448.log 2024-04-25T04:46:03.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146253.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146253.log.gz 2024-04-25T04:46:03.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138048.log 2024-04-25T04:46:03.982 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121448.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121448.log.gz 2024-04-25T04:46:03.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99796.log 2024-04-25T04:46:03.983 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138048.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138048.log.gz 2024-04-25T04:46:03.983 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42811.log 2024-04-25T04:46:03.984 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99796.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99796.log.gz 2024-04-25T04:46:03.984 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111356.log 2024-04-25T04:46:03.985 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42811.log.gz 2024-04-25T04:46:03.985 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108124.log 2024-04-25T04:46:03.986 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111356.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111356.log.gz 2024-04-25T04:46:03.986 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116976.log 2024-04-25T04:46:03.986 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108124.log.gz 2024-04-25T04:46:03.987 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147647.log 2024-04-25T04:46:03.987 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116976.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116976.log.gz 2024-04-25T04:46:03.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63755.log 2024-04-25T04:46:03.988 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147647.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147647.log.gz 2024-04-25T04:46:03.989 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155017.log 2024-04-25T04:46:03.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63755.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63755.log.gz 2024-04-25T04:46:03.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157253.log 2024-04-25T04:46:03.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155017.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155017.log.gz 2024-04-25T04:46:03.991 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95690.log 2024-04-25T04:46:03.991 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157253.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157253.log.gz 2024-04-25T04:46:03.991 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139728.log 2024-04-25T04:46:03.992 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95690.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95690.log.gz 2024-04-25T04:46:03.992 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141656.log 2024-04-25T04:46:03.993 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139728.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139728.log.gz 2024-04-25T04:46:03.993 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47760.log 2024-04-25T04:46:03.994 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141656.log.gz 2024-04-25T04:46:03.994 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57998.log 2024-04-25T04:46:03.995 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47760.log.gz 2024-04-25T04:46:03.995 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77467.log 2024-04-25T04:46:03.996 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57998.log.gz 2024-04-25T04:46:03.996 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58579.log 2024-04-25T04:46:03.996 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77467.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77467.log.gz 2024-04-25T04:46:03.997 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49913.log 2024-04-25T04:46:03.997 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58579.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58579.log.gz 2024-04-25T04:46:03.998 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45268.log 2024-04-25T04:46:03.998 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49913.log.gz 2024-04-25T04:46:03.999 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47671.log 2024-04-25T04:46:03.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45268.log.gz 2024-04-25T04:46:04.000 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122487.log 2024-04-25T04:46:04.000 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47671.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47671.log.gz 2024-04-25T04:46:04.000 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51775.log 2024-04-25T04:46:04.001 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122487.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122487.log.gz 2024-04-25T04:46:04.001 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113409.log 2024-04-25T04:46:04.002 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51775.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51775.log.gz 2024-04-25T04:46:04.002 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125615.log 2024-04-25T04:46:04.003 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113409.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113409.log.gz 2024-04-25T04:46:04.003 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98828.log 2024-04-25T04:46:04.004 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125615.log.gz 2024-04-25T04:46:04.004 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114321.log 2024-04-25T04:46:04.004 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98828.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98828.log.gz 2024-04-25T04:46:04.005 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112257.log 2024-04-25T04:46:04.006 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114321.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114321.log.gz 2024-04-25T04:46:04.006 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65253.log 2024-04-25T04:46:04.006 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112257.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112257.log.gz 2024-04-25T04:46:04.007 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67032.log 2024-04-25T04:46:04.007 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65253.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65253.log.gz 2024-04-25T04:46:04.007 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81689.log 2024-04-25T04:46:04.008 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67032.log.gz 2024-04-25T04:46:04.008 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78145.log 2024-04-25T04:46:04.009 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81689.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81689.log.gz 2024-04-25T04:46:04.009 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112810.log 2024-04-25T04:46:04.010 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78145.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78145.log.gz 2024-04-25T04:46:04.010 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160478.log 2024-04-25T04:46:04.011 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157917.log 2024-04-25T04:46:04.011 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112810.log.gz 2024-04-25T04:46:04.011 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160478.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160478.log.gz 2024-04-25T04:46:04.011 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94803.log 2024-04-25T04:46:04.012 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157917.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157917.log.gz 2024-04-25T04:46:04.012 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.113000.log 2024-04-25T04:46:04.013 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94803.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94803.log.gz 2024-04-25T04:46:04.013 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113656.log 2024-04-25T04:46:04.014 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.113000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.113000.log.gz 2024-04-25T04:46:04.014 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118385.log 2024-04-25T04:46:04.015 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113656.log.gz 2024-04-25T04:46:04.015 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155648.log 2024-04-25T04:46:04.016 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118385.log.gz 2024-04-25T04:46:04.016 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153771.log 2024-04-25T04:46:04.017 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155648.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155648.log.gz 2024-04-25T04:46:04.017 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96947.log 2024-04-25T04:46:04.017 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153771.log.gz 2024-04-25T04:46:04.018 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136766.log 2024-04-25T04:46:04.018 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96947.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96947.log.gz 2024-04-25T04:46:04.019 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64225.log 2024-04-25T04:46:04.019 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136766.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136766.log.gz 2024-04-25T04:46:04.019 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65966.log 2024-04-25T04:46:04.020 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.64225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64225.log.gz 2024-04-25T04:46:04.020 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67497.log 2024-04-25T04:46:04.021 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.65966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65966.log.gz 2024-04-25T04:46:04.021 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77353.log 2024-04-25T04:46:04.022 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67497.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67497.log.gz 2024-04-25T04:46:04.022 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90344.log 2024-04-25T04:46:04.023 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77353.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77353.log.gz 2024-04-25T04:46:04.023 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99981.log 2024-04-25T04:46:04.023 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90344.log.gz 2024-04-25T04:46:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96340.log 2024-04-25T04:46:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99981.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99981.log.gz 2024-04-25T04:46:04.024 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116190.log 2024-04-25T04:46:04.025 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96340.log.gz 2024-04-25T04:46:04.025 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139369.log 2024-04-25T04:46:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116190.log.gz 2024-04-25T04:46:04.026 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128848.log 2024-04-25T04:46:04.027 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139369.log.gz 2024-04-25T04:46:04.027 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138910.log 2024-04-25T04:46:04.028 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128848.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128848.log.gz 2024-04-25T04:46:04.028 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135362.log 2024-04-25T04:46:04.029 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138910.log.gz 2024-04-25T04:46:04.029 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85422.log 2024-04-25T04:46:04.030 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135362.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135362.log.gz 2024-04-25T04:46:04.030 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101161.log 2024-04-25T04:46:04.030 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85422.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85422.log.gz 2024-04-25T04:46:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49385.log 2024-04-25T04:46:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101161.log.gz 2024-04-25T04:46:04.031 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98962.log 2024-04-25T04:46:04.032 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49385.log.gz 2024-04-25T04:46:04.032 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96627.log 2024-04-25T04:46:04.033 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98962.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98962.log.gz 2024-04-25T04:46:04.033 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53261.log 2024-04-25T04:46:04.034 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96627.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96627.log.gz 2024-04-25T04:46:04.034 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76983.log 2024-04-25T04:46:04.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53261.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53261.log.gz 2024-04-25T04:46:04.035 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135574.log 2024-04-25T04:46:04.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.76983.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76983.log.gz 2024-04-25T04:46:04.036 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56563.log 2024-04-25T04:46:04.036 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135574.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135574.log.gz 2024-04-25T04:46:04.036 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52003.log 2024-04-25T04:46:04.037 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56563.log.gz 2024-04-25T04:46:04.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111432.log 2024-04-25T04:46:04.038 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52003.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52003.log.gz 2024-04-25T04:46:04.038 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82767.log 2024-04-25T04:46:04.039 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111432.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111432.log.gz 2024-04-25T04:46:04.039 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.100864.log 2024-04-25T04:46:04.040 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82767.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82767.log.gz 2024-04-25T04:46:04.040 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84936.log 2024-04-25T04:46:04.040 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.100864.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.100864.log.gz 2024-04-25T04:46:04.041 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112470.log 2024-04-25T04:46:04.041 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84936.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84936.log.gz 2024-04-25T04:46:04.042 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123052.log 2024-04-25T04:46:04.042 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112470.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112470.log.gz 2024-04-25T04:46:04.042 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62348.log 2024-04-25T04:46:04.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123052.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123052.log.gz 2024-04-25T04:46:04.043 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80507.log 2024-04-25T04:46:04.044 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62348.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62348.log.gz 2024-04-25T04:46:04.044 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46929.log 2024-04-25T04:46:04.045 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80507.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80507.log.gz 2024-04-25T04:46:04.045 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51687.log 2024-04-25T04:46:04.045 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46929.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46929.log.gz 2024-04-25T04:46:04.046 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105474.log 2024-04-25T04:46:04.046 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51687.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51687.log.gz 2024-04-25T04:46:04.047 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89920.log 2024-04-25T04:46:04.047 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105474.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105474.log.gz 2024-04-25T04:46:04.048 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153100.log 2024-04-25T04:46:04.048 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.89920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46154.log 2024-04-25T04:46:04.048 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89920.log.gz 2024-04-25T04:46:04.049 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153100.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153100.log.gz 2024-04-25T04:46:04.049 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56791.log 2024-04-25T04:46:04.050 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46154.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46154.log.gz 2024-04-25T04:46:04.050 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137456.log 2024-04-25T04:46:04.051 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56791.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56791.log.gz 2024-04-25T04:46:04.051 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63050.log 2024-04-25T04:46:04.051 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137456.log.gz 2024-04-25T04:46:04.052 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65758.log 2024-04-25T04:46:04.052 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63050.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63050.log.gz 2024-04-25T04:46:04.053 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100233.log 2024-04-25T04:46:04.053 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65758.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65758.log.gz 2024-04-25T04:46:04.054 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61788.log 2024-04-25T04:46:04.054 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100233.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100233.log.gz 2024-04-25T04:46:04.054 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92293.log 2024-04-25T04:46:04.055 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61788.log.gz 2024-04-25T04:46:04.055 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144268.log 2024-04-25T04:46:04.056 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92293.log.gz 2024-04-25T04:46:04.056 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115499.log 2024-04-25T04:46:04.057 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144268.log.gz 2024-04-25T04:46:04.057 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102588.log 2024-04-25T04:46:04.058 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115499.log.gz 2024-04-25T04:46:04.058 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82723.log 2024-04-25T04:46:04.058 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102588.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102588.log.gz 2024-04-25T04:46:04.059 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140028.log 2024-04-25T04:46:04.059 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82723.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82723.log.gz 2024-04-25T04:46:04.060 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58964.log 2024-04-25T04:46:04.060 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140028.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140028.log.gz 2024-04-25T04:46:04.061 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157593.log 2024-04-25T04:46:04.062 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58964.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58964.log.gz 2024-04-25T04:46:04.062 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100231.log 2024-04-25T04:46:04.062 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157593.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157593.log.gz 2024-04-25T04:46:04.062 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45754.log 2024-04-25T04:46:04.063 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100231.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100231.log.gz 2024-04-25T04:46:04.063 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157439.log 2024-04-25T04:46:04.064 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45754.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45754.log.gz 2024-04-25T04:46:04.064 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94113.log 2024-04-25T04:46:04.065 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157439.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157439.log.gz 2024-04-25T04:46:04.065 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139684.log 2024-04-25T04:46:04.066 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94113.log.gz 2024-04-25T04:46:04.066 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137236.log 2024-04-25T04:46:04.067 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139684.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139684.log.gz 2024-04-25T04:46:04.067 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84875.log 2024-04-25T04:46:04.068 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137236.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137236.log.gz 2024-04-25T04:46:04.068 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53742.log 2024-04-25T04:46:04.068 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84875.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84875.log.gz 2024-04-25T04:46:04.069 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119153.log 2024-04-25T04:46:04.069 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53742.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53742.log.gz 2024-04-25T04:46:04.069 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93305.log 2024-04-25T04:46:04.070 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119153.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119153.log.gz 2024-04-25T04:46:04.070 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61423.log 2024-04-25T04:46:04.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93305.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93305.log.gz 2024-04-25T04:46:04.071 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124697.log 2024-04-25T04:46:04.072 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61423.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61423.log.gz 2024-04-25T04:46:04.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78653.log 2024-04-25T04:46:04.073 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124697.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124697.log.gz 2024-04-25T04:46:04.073 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123756.log 2024-04-25T04:46:04.073 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.78653.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78653.log.gz 2024-04-25T04:46:04.074 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62210.log 2024-04-25T04:46:04.075 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.123756.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123756.log.gz 2024-04-25T04:46:04.075 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84580.log 2024-04-25T04:46:04.075 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62210.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62210.log.gz 2024-04-25T04:46:04.075 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60149.log 2024-04-25T04:46:04.076 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84580.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84580.log.gz 2024-04-25T04:46:04.076 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116770.log 2024-04-25T04:46:04.077 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60149.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60149.log.gz 2024-04-25T04:46:04.077 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45627.log 2024-04-25T04:46:04.078 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116770.log.gz 2024-04-25T04:46:04.078 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73290.log 2024-04-25T04:46:04.079 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45627.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45627.log.gz 2024-04-25T04:46:04.079 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130091.log 2024-04-25T04:46:04.080 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73290.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73290.log.gz 2024-04-25T04:46:04.080 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126959.log 2024-04-25T04:46:04.081 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130091.log.gz 2024-04-25T04:46:04.081 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95133.log 2024-04-25T04:46:04.081 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.126959.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126959.log.gz 2024-04-25T04:46:04.082 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144823.log 2024-04-25T04:46:04.082 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95133.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95133.log.gz 2024-04-25T04:46:04.087 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131288.log 2024-04-25T04:46:04.087 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144823.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144823.log.gz 2024-04-25T04:46:04.087 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132068.log 2024-04-25T04:46:04.087 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131288.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131288.log.gz 2024-04-25T04:46:04.087 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120159.log 2024-04-25T04:46:04.087 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132068.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132068.log.gz 2024-04-25T04:46:04.087 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71015.log 2024-04-25T04:46:04.088 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120159.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120159.log.gz 2024-04-25T04:46:04.088 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56351.log 2024-04-25T04:46:04.088 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71015.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71015.log.gz 2024-04-25T04:46:04.089 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145611.log 2024-04-25T04:46:04.089 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56351.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56351.log.gz 2024-04-25T04:46:04.089 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139986.log 2024-04-25T04:46:04.089 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145611.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145611.log.gz 2024-04-25T04:46:04.089 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100523.log 2024-04-25T04:46:04.090 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139986.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139986.log.gz 2024-04-25T04:46:04.090 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125883.log 2024-04-25T04:46:04.091 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100523.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/cluster2-client.admin.100523.log.gz-- /var/log/ceph/cluster2-client.mirror.77634.log 2024-04-25T04:46:04.091 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-25T04:46:04.092 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125883.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125883.log.gz 2024-04-25T04:46:04.092 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160796.log 2024-04-25T04:46:04.092 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77634.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77634.log.gz 2024-04-25T04:46:04.092 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72576.log 2024-04-25T04:46:04.093 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160796.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160796.log.gz 2024-04-25T04:46:04.093 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93963.log 2024-04-25T04:46:04.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72576.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72576.log.gz 2024-04-25T04:46:04.094 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57451.log 2024-04-25T04:46:04.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93963.log.gz 2024-04-25T04:46:04.095 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44461.log 2024-04-25T04:46:04.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57451.log.gz 2024-04-25T04:46:04.096 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130672.log 2024-04-25T04:46:04.096 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44461.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44461.log.gz 2024-04-25T04:46:04.097 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159693.log 2024-04-25T04:46:04.097 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130672.log.gz 2024-04-25T04:46:04.097 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85623.log 2024-04-25T04:46:04.098 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.159693.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.159693.log.gz 2024-04-25T04:46:04.098 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93645.log 2024-04-25T04:46:04.099 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85623.log.gz 2024-04-25T04:46:04.099 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47630.log 2024-04-25T04:46:04.100 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93645.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93645.log.gz 2024-04-25T04:46:04.100 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101925.log 2024-04-25T04:46:04.101 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47630.log.gz 2024-04-25T04:46:04.101 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92394.log 2024-04-25T04:46:04.102 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101925.log.gz 2024-04-25T04:46:04.102 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108130.log 2024-04-25T04:46:04.103 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92394.log.gz 2024-04-25T04:46:04.103 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158294.log 2024-04-25T04:46:04.103 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108130.log.gz 2024-04-25T04:46:04.104 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53689.log 2024-04-25T04:46:04.104 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158294.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158294.log.gz 2024-04-25T04:46:04.104 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64056.log 2024-04-25T04:46:04.105 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53689.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53689.log.gz 2024-04-25T04:46:04.105 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44039.log 2024-04-25T04:46:04.106 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64056.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64056.log.gz 2024-04-25T04:46:04.106 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104203.log 2024-04-25T04:46:04.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44039.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44039.log.gz 2024-04-25T04:46:04.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126783.log 2024-04-25T04:46:04.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104203.log.gz 2024-04-25T04:46:04.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142515.log 2024-04-25T04:46:04.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.126783.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126783.log.gz 2024-04-25T04:46:04.109 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84264.log 2024-04-25T04:46:04.109 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142515.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142515.log.gz 2024-04-25T04:46:04.110 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81525.log 2024-04-25T04:46:04.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84264.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84264.log.gz 2024-04-25T04:46:04.111 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92057.log 2024-04-25T04:46:04.111 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81525.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81525.log.gz 2024-04-25T04:46:04.111 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119556.log 2024-04-25T04:46:04.112 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92057.log.gz 2024-04-25T04:46:04.112 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83353.log 2024-04-25T04:46:04.113 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119556.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119556.log.gz 2024-04-25T04:46:04.113 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83247.log 2024-04-25T04:46:04.114 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83353.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83353.log.gz 2024-04-25T04:46:04.114 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158557.log 2024-04-25T04:46:04.115 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83247.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83247.log.gz 2024-04-25T04:46:04.115 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121462.log 2024-04-25T04:46:04.115 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158557.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158557.log.gz 2024-04-25T04:46:04.116 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138298.log 2024-04-25T04:46:04.117 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121462.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121462.log.gz 2024-04-25T04:46:04.117 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46931.log 2024-04-25T04:46:04.117 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138298.log.gz 2024-04-25T04:46:04.118 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75745.log 2024-04-25T04:46:04.118 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46931.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46931.log.gz 2024-04-25T04:46:04.119 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155356.log 2024-04-25T04:46:04.119 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75745.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75745.log.gz 2024-04-25T04:46:04.120 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100952.log 2024-04-25T04:46:04.120 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155356.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155356.log.gz 2024-04-25T04:46:04.121 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45056.log 2024-04-25T04:46:04.121 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100952.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100952.log.gz 2024-04-25T04:46:04.121 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61870.log 2024-04-25T04:46:04.122 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45056.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45056.log.gz 2024-04-25T04:46:04.122 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75362.log 2024-04-25T04:46:04.123 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61870.log.gz 2024-04-25T04:46:04.123 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108397.log 2024-04-25T04:46:04.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75362.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75362.log.gz 2024-04-25T04:46:04.124 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43830.log 2024-04-25T04:46:04.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108397.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108397.log.gz 2024-04-25T04:46:04.125 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94315.log 2024-04-25T04:46:04.125 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43830.log.gz 2024-04-25T04:46:04.126 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142075.log 2024-04-25T04:46:04.127 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94315.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94315.log.gz 2024-04-25T04:46:04.127 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79964.log 2024-04-25T04:46:04.128 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142075.log.gz 2024-04-25T04:46:04.128 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91273.log 2024-04-25T04:46:04.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79964.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79964.log.gz 2024-04-25T04:46:04.129 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47455.log 2024-04-25T04:46:04.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91273.log.gz 2024-04-25T04:46:04.129 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73947.log 2024-04-25T04:46:04.130 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47455.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47455.log.gz 2024-04-25T04:46:04.130 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48785.log 2024-04-25T04:46:04.131 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.73947.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73947.log.gz 2024-04-25T04:46:04.131 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141296.log 2024-04-25T04:46:04.132 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48785.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48785.log.gz 2024-04-25T04:46:04.132 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68091.log 2024-04-25T04:46:04.133 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141296.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141296.log.gz 2024-04-25T04:46:04.133 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63226.log 2024-04-25T04:46:04.134 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68091.log.gz 2024-04-25T04:46:04.134 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60567.log 2024-04-25T04:46:04.135 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63226.log.gz 2024-04-25T04:46:04.135 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91769.log 2024-04-25T04:46:04.135 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60567.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60567.log.gz 2024-04-25T04:46:04.136 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99760.log 2024-04-25T04:46:04.136 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91769.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91769.log.gz 2024-04-25T04:46:04.136 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45857.log 2024-04-25T04:46:04.137 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99760.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99760.log.gz 2024-04-25T04:46:04.137 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152154.log 2024-04-25T04:46:04.138 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45857.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45857.log.gz 2024-04-25T04:46:04.138 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157684.log 2024-04-25T04:46:04.139 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152154.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152154.log.gz 2024-04-25T04:46:04.139 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47227.log 2024-04-25T04:46:04.140 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157684.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157684.log.gz 2024-04-25T04:46:04.140 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121023.log 2024-04-25T04:46:04.141 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47227.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47227.log.gz 2024-04-25T04:46:04.141 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112676.log 2024-04-25T04:46:04.142 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121023.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121023.log.gz 2024-04-25T04:46:04.142 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151924.log 2024-04-25T04:46:04.142 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112676.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112676.log.gz 2024-04-25T04:46:04.142 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126017.log 2024-04-25T04:46:04.143 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151924.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151924.log.gz 2024-04-25T04:46:04.143 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120201.log 2024-04-25T04:46:04.144 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126017.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126017.log.gz 2024-04-25T04:46:04.144 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86599.log 2024-04-25T04:46:04.145 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120201.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120201.log.gz 2024-04-25T04:46:04.145 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99193.log 2024-04-25T04:46:04.145 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86599.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86599.log.gz 2024-04-25T04:46:04.146 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50567.log 2024-04-25T04:46:04.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99193.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99193.log.gz 2024-04-25T04:46:04.147 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65109.log 2024-04-25T04:46:04.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50567.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50567.log.gz 2024-04-25T04:46:04.148 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107934.log 2024-04-25T04:46:04.148 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65109.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65109.log.gz 2024-04-25T04:46:04.148 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124340.log 2024-04-25T04:46:04.149 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107934.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107934.log.gz 2024-04-25T04:46:04.149 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135023.log 2024-04-25T04:46:04.150 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124340.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124340.log.gz 2024-04-25T04:46:04.150 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74071.log 2024-04-25T04:46:04.151 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135023.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135023.log.gz 2024-04-25T04:46:04.151 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139474.log 2024-04-25T04:46:04.152 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74071.log.gz 2024-04-25T04:46:04.152 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67880.log 2024-04-25T04:46:04.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139474.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139474.log.gz 2024-04-25T04:46:04.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63805.log 2024-04-25T04:46:04.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67880.log.gz 2024-04-25T04:46:04.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58374.log 2024-04-25T04:46:04.155 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.63805.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63805.log.gz 2024-04-25T04:46:04.155 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74740.log 2024-04-25T04:46:04.156 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58374.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58374.log.gz 2024-04-25T04:46:04.156 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59449.log 2024-04-25T04:46:04.156 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74740.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74740.log.gz 2024-04-25T04:46:04.157 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117186.log 2024-04-25T04:46:04.157 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59449.log.gz 2024-04-25T04:46:04.157 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140029.log 2024-04-25T04:46:04.158 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117186.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117186.log.gz 2024-04-25T04:46:04.158 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61304.log 2024-04-25T04:46:04.159 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140029.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140029.log.gz 2024-04-25T04:46:04.159 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66100.log 2024-04-25T04:46:04.160 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61304.log.gz 2024-04-25T04:46:04.160 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45166.log 2024-04-25T04:46:04.161 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66100.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66100.log.gz 2024-04-25T04:46:04.161 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106357.log 2024-04-25T04:46:04.161 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45166.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45166.log.gz 2024-04-25T04:46:04.162 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136631.log 2024-04-25T04:46:04.162 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106357.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106357.log.gz 2024-04-25T04:46:04.163 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131229.log 2024-04-25T04:46:04.163 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136631.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136631.log.gz 2024-04-25T04:46:04.163 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96180.log 2024-04-25T04:46:04.164 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131229.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131229.log.gz 2024-04-25T04:46:04.164 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59681.log 2024-04-25T04:46:04.165 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96180.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96180.log.gz 2024-04-25T04:46:04.165 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81061.log 2024-04-25T04:46:04.166 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59681.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59681.log.gz 2024-04-25T04:46:04.166 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72534.log 2024-04-25T04:46:04.167 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81061.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81061.log.gz 2024-04-25T04:46:04.167 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130299.log 2024-04-25T04:46:04.167 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72534.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72534.log.gz 2024-04-25T04:46:04.168 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131125.log 2024-04-25T04:46:04.168 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130299.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130299.log.gz 2024-04-25T04:46:04.168 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115439.log 2024-04-25T04:46:04.169 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131125.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131125.log.gz 2024-04-25T04:46:04.169 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91918.log 2024-04-25T04:46:04.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115439.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115439.log.gz 2024-04-25T04:46:04.170 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143496.log 2024-04-25T04:46:04.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91918.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91918.log.gz 2024-04-25T04:46:04.171 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85924.log 2024-04-25T04:46:04.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143496.log.gz 2024-04-25T04:46:04.172 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95231.log 2024-04-25T04:46:04.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85924.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85924.log.gz 2024-04-25T04:46:04.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110775.log 2024-04-25T04:46:04.173 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95231.log.gz 2024-04-25T04:46:04.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97587.log 2024-04-25T04:46:04.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110775.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110775.log.gz 2024-04-25T04:46:04.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104496.log 2024-04-25T04:46:04.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97587.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97587.log.gz 2024-04-25T04:46:04.175 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80405.log 2024-04-25T04:46:04.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104496.log.gz 2024-04-25T04:46:04.176 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144750.log 2024-04-25T04:46:04.177 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80405.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80405.log.gz 2024-04-25T04:46:04.177 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74958.log 2024-04-25T04:46:04.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144750.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144750.log.gz 2024-04-25T04:46:04.178 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121063.log 2024-04-25T04:46:04.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74958.log.gz 2024-04-25T04:46:04.179 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118949.log 2024-04-25T04:46:04.179 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121063.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121063.log.gz 2024-04-25T04:46:04.180 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49942.log 2024-04-25T04:46:04.180 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118949.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118949.log.gz 2024-04-25T04:46:04.181 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86819.log 2024-04-25T04:46:04.181 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.49942.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49942.log.gz 2024-04-25T04:46:04.181 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57512.log 2024-04-25T04:46:04.182 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86819.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86819.log.gz 2024-04-25T04:46:04.182 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64871.log 2024-04-25T04:46:04.183 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57512.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57512.log.gz 2024-04-25T04:46:04.183 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100811.log 2024-04-25T04:46:04.184 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64871.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64871.log.gz 2024-04-25T04:46:04.184 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71309.log 2024-04-25T04:46:04.185 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100811.log.gz 2024-04-25T04:46:04.185 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47696.log 2024-04-25T04:46:04.186 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71309.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71309.log.gz 2024-04-25T04:46:04.186 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159568.log 2024-04-25T04:46:04.186 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47696.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47696.log.gz 2024-04-25T04:46:04.186 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105385.log 2024-04-25T04:46:04.187 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159568.log.gz 2024-04-25T04:46:04.187 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138361.log 2024-04-25T04:46:04.188 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105385.log.gz 2024-04-25T04:46:04.188 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47456.log 2024-04-25T04:46:04.189 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138361.log.gz 2024-04-25T04:46:04.189 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138760.log 2024-04-25T04:46:04.190 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47456.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47456.log.gz 2024-04-25T04:46:04.190 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45794.log 2024-04-25T04:46:04.190 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138760.log.gz 2024-04-25T04:46:04.191 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102966.log 2024-04-25T04:46:04.191 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45794.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45794.log.gz 2024-04-25T04:46:04.192 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151075.log 2024-04-25T04:46:04.192 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102966.log.gz 2024-04-25T04:46:04.193 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115438.log 2024-04-25T04:46:04.194 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151075.log.gz 2024-04-25T04:46:04.194 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54239.log 2024-04-25T04:46:04.194 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115438.log.gz 2024-04-25T04:46:04.195 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46488.log 2024-04-25T04:46:04.195 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54239.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54239.log.gz 2024-04-25T04:46:04.195 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52592.log 2024-04-25T04:46:04.196 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46488.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46488.log.gz 2024-04-25T04:46:04.196 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94292.log 2024-04-25T04:46:04.197 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52592.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52592.log.gz 2024-04-25T04:46:04.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115046.log 2024-04-25T04:46:04.198 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94292.log.gz 2024-04-25T04:46:04.198 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112147.log 2024-04-25T04:46:04.199 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115046.log.gz 2024-04-25T04:46:04.199 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118195.log 2024-04-25T04:46:04.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112147.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112147.log.gz 2024-04-25T04:46:04.200 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110683.log 2024-04-25T04:46:04.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118195.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118195.log.gz 2024-04-25T04:46:04.201 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134268.log 2024-04-25T04:46:04.201 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110683.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110683.log.gz 2024-04-25T04:46:04.202 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96172.log 2024-04-25T04:46:04.202 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134268.log.gz 2024-04-25T04:46:04.203 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133226.log 2024-04-25T04:46:04.203 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96172.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96172.log.gz 2024-04-25T04:46:04.203 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64582.log 2024-04-25T04:46:04.204 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133226.log.gz 2024-04-25T04:46:04.204 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72913.log 2024-04-25T04:46:04.205 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64582.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64582.log.gz 2024-04-25T04:46:04.205 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129712.log 2024-04-25T04:46:04.206 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72913.log.gz 2024-04-25T04:46:04.206 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160344.log 2024-04-25T04:46:04.206 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129712.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129712.log.gz 2024-04-25T04:46:04.207 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112227.log 2024-04-25T04:46:04.208 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160344.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160344.log.gz 2024-04-25T04:46:04.208 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43275.log 2024-04-25T04:46:04.208 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112227.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112227.log.gz 2024-04-25T04:46:04.209 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106438.log 2024-04-25T04:46:04.209 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43275.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43275.log.gz 2024-04-25T04:46:04.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125302.log 2024-04-25T04:46:04.210 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106438.log.gz 2024-04-25T04:46:04.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126894.log 2024-04-25T04:46:04.211 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125302.log.gz 2024-04-25T04:46:04.211 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48116.log 2024-04-25T04:46:04.212 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126894.log.gz 2024-04-25T04:46:04.212 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122680.log 2024-04-25T04:46:04.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48116.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48116.log.gz 2024-04-25T04:46:04.213 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95981.log 2024-04-25T04:46:04.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122680.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122680.log.gz 2024-04-25T04:46:04.214 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54486.log 2024-04-25T04:46:04.214 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95981.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95981.log.gz 2024-04-25T04:46:04.215 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148787.log 2024-04-25T04:46:04.215 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54486.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54486.log.gz 2024-04-25T04:46:04.216 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60066.log 2024-04-25T04:46:04.216 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148787.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148787.log.gz 2024-04-25T04:46:04.216 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89521.log 2024-04-25T04:46:04.217 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60066.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60066.log.gz 2024-04-25T04:46:04.217 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154516.log 2024-04-25T04:46:04.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89521.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89521.log.gz 2024-04-25T04:46:04.218 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133095.log 2024-04-25T04:46:04.219 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154516.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154516.log.gz 2024-04-25T04:46:04.219 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73911.log 2024-04-25T04:46:04.220 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133095.log.gz 2024-04-25T04:46:04.220 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75883.log 2024-04-25T04:46:04.220 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.73911.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.73911.log.gz 2024-04-25T04:46:04.221 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127603.log 2024-04-25T04:46:04.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75883.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75883.log.gz 2024-04-25T04:46:04.222 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124542.log 2024-04-25T04:46:04.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127603.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127603.log.gz 2024-04-25T04:46:04.223 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157813.log 2024-04-25T04:46:04.223 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124542.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124542.log.gz 2024-04-25T04:46:04.223 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51568.log 2024-04-25T04:46:04.224 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157813.log.gz 2024-04-25T04:46:04.224 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138700.log 2024-04-25T04:46:04.225 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51568.log.gz 2024-04-25T04:46:04.225 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60505.log 2024-04-25T04:46:04.226 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138700.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138700.log.gz 2024-04-25T04:46:04.226 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83994.log 2024-04-25T04:46:04.226 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60505.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60505.log.gz 2024-04-25T04:46:04.227 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131919.log 2024-04-25T04:46:04.227 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83994.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83994.log.gz 2024-04-25T04:46:04.228 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110857.log 2024-04-25T04:46:04.228 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131919.log.gz 2024-04-25T04:46:04.228 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55968.log 2024-04-25T04:46:04.229 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110857.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110857.log.gz 2024-04-25T04:46:04.229 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80951.log 2024-04-25T04:46:04.230 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55968.log.gz 2024-04-25T04:46:04.230 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56141.log 2024-04-25T04:46:04.231 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80951.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80951.log.gz 2024-04-25T04:46:04.231 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57704.log 2024-04-25T04:46:04.232 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56141.log.gz 2024-04-25T04:46:04.232 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75686.log 2024-04-25T04:46:04.233 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57704.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57704.log.gz 2024-04-25T04:46:04.233 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135289.log 2024-04-25T04:46:04.234 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75686.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75686.log.gz 2024-04-25T04:46:04.234 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51856.log 2024-04-25T04:46:04.235 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135289.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135289.log.gz 2024-04-25T04:46:04.235 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108877.log 2024-04-25T04:46:04.236 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51856.log.gz 2024-04-25T04:46:04.236 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156090.log 2024-04-25T04:46:04.237 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108877.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108877.log.gz 2024-04-25T04:46:04.237 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103698.log 2024-04-25T04:46:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156090.log.gz 2024-04-25T04:46:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96318.log 2024-04-25T04:46:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103698.log.gz 2024-04-25T04:46:04.238 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148890.log 2024-04-25T04:46:04.239 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96318.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96318.log.gz 2024-04-25T04:46:04.239 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79839.log 2024-04-25T04:46:04.240 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148890.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148890.log.gz 2024-04-25T04:46:04.240 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122596.log 2024-04-25T04:46:04.241 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79839.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79839.log.gz 2024-04-25T04:46:04.241 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107490.log 2024-04-25T04:46:04.242 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122596.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122596.log.gz 2024-04-25T04:46:04.242 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69273.log 2024-04-25T04:46:04.243 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107490.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107490.log.gz 2024-04-25T04:46:04.243 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111653.log 2024-04-25T04:46:04.244 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69273.log.gz 2024-04-25T04:46:04.244 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62815.log 2024-04-25T04:46:04.245 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111653.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111653.log.gz 2024-04-25T04:46:04.245 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88481.log 2024-04-25T04:46:04.245 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62815.log.gz 2024-04-25T04:46:04.246 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57872.log 2024-04-25T04:46:04.246 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88481.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88481.log.gz 2024-04-25T04:46:04.246 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110397.log 2024-04-25T04:46:04.247 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57872.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57872.log.gz 2024-04-25T04:46:04.247 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44317.log 2024-04-25T04:46:04.248 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110397.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110397.log.gz 2024-04-25T04:46:04.248 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77695.log 2024-04-25T04:46:04.249 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44317.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44317.log.gz 2024-04-25T04:46:04.249 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110286.log 2024-04-25T04:46:04.250 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77695.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77695.log.gz 2024-04-25T04:46:04.250 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135578.log 2024-04-25T04:46:04.250 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110286.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110286.log.gz 2024-04-25T04:46:04.251 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79544.log 2024-04-25T04:46:04.251 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135578.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135578.log.gz 2024-04-25T04:46:04.252 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128169.log 2024-04-25T04:46:04.252 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79544.log.gz 2024-04-25T04:46:04.252 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131647.log 2024-04-25T04:46:04.253 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128169.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128169.log.gz 2024-04-25T04:46:04.253 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96552.log 2024-04-25T04:46:04.254 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131647.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131647.log.gz 2024-04-25T04:46:04.254 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61012.log 2024-04-25T04:46:04.255 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96552.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96552.log.gz 2024-04-25T04:46:04.255 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152722.log 2024-04-25T04:46:04.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61012.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61012.log.gz 2024-04-25T04:46:04.256 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109531.log 2024-04-25T04:46:04.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152722.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152722.log.gz 2024-04-25T04:46:04.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53331.log 2024-04-25T04:46:04.257 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109531.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109531.log.gz 2024-04-25T04:46:04.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120875.log 2024-04-25T04:46:04.258 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53331.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53331.log.gz 2024-04-25T04:46:04.258 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49294.log 2024-04-25T04:46:04.259 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120875.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120875.log.gz 2024-04-25T04:46:04.259 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80668.log 2024-04-25T04:46:04.260 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49294.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49294.log.gz 2024-04-25T04:46:04.260 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69097.log 2024-04-25T04:46:04.261 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80668.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80668.log.gz 2024-04-25T04:46:04.261 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85194.log 2024-04-25T04:46:04.261 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69097.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69097.log.gz 2024-04-25T04:46:04.262 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83477.log 2024-04-25T04:46:04.262 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85194.log.gz 2024-04-25T04:46:04.263 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86820.log 2024-04-25T04:46:04.263 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83477.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83477.log.gz 2024-04-25T04:46:04.264 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46406.log 2024-04-25T04:46:04.264 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86820.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86820.log.gz 2024-04-25T04:46:04.265 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140501.log 2024-04-25T04:46:04.265 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46406.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46406.log.gz 2024-04-25T04:46:04.265 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82471.log 2024-04-25T04:46:04.266 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140501.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140501.log.gz 2024-04-25T04:46:04.266 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103993.log 2024-04-25T04:46:04.267 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82471.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82471.log.gz 2024-04-25T04:46:04.267 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74492.log 2024-04-25T04:46:04.268 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103993.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103993.log.gz 2024-04-25T04:46:04.268 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121437.log 2024-04-25T04:46:04.269 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74492.log.gz 2024-04-25T04:46:04.269 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128418.log 2024-04-25T04:46:04.269 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121437.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121437.log.gz 2024-04-25T04:46:04.270 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.110937.log 2024-04-25T04:46:04.270 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128418.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128418.log.gz 2024-04-25T04:46:04.270 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108948.log 2024-04-25T04:46:04.271 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.110937.log: gzip 0.0%-5 -- replaced with /var/log/ceph/cluster2-client.admin.110937.log.gz--verbose 2024-04-25T04:46:04.271 INFO:teuthology.orchestra.run.smithi188.stderr:-- /var/log/ceph/cluster2-client.mirror.146942.log 2024-04-25T04:46:04.272 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108948.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108948.log.gz 2024-04-25T04:46:04.272 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64548.log 2024-04-25T04:46:04.273 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.146942.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.146942.log.gz 2024-04-25T04:46:04.273 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140752.log 2024-04-25T04:46:04.274 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64548.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64548.log.gz 2024-04-25T04:46:04.274 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145831.log 2024-04-25T04:46:04.275 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140752.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140752.log.gz 2024-04-25T04:46:04.275 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119505.log 2024-04-25T04:46:04.276 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145831.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145831.log.gz 2024-04-25T04:46:04.276 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72285.log 2024-04-25T04:46:04.277 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119505.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119505.log.gz 2024-04-25T04:46:04.277 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155188.log 2024-04-25T04:46:04.278 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72285.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72285.log.gz 2024-04-25T04:46:04.278 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91624.log 2024-04-25T04:46:04.278 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155188.log.gz 2024-04-25T04:46:04.279 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112623.log 2024-04-25T04:46:04.279 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91624.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91624.log.gz 2024-04-25T04:46:04.279 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144944.log 2024-04-25T04:46:04.280 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112623.log.gz 2024-04-25T04:46:04.280 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104078.log 2024-04-25T04:46:04.281 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144944.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144944.log.gz 2024-04-25T04:46:04.281 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66127.log 2024-04-25T04:46:04.282 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104078.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104078.log.gz 2024-04-25T04:46:04.282 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126898.log 2024-04-25T04:46:04.283 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66127.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66127.log.gz 2024-04-25T04:46:04.283 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57397.log 2024-04-25T04:46:04.283 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126898.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126898.log.gz 2024-04-25T04:46:04.284 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68133.log 2024-04-25T04:46:04.284 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57397.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57397.log.gz 2024-04-25T04:46:04.284 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123286.log 2024-04-25T04:46:04.285 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68133.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68133.log.gz 2024-04-25T04:46:04.285 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140540.log 2024-04-25T04:46:04.286 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123286.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123286.log.gz 2024-04-25T04:46:04.286 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66364.log 2024-04-25T04:46:04.287 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140540.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140540.log.gz 2024-04-25T04:46:04.287 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68937.log 2024-04-25T04:46:04.288 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66364.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66364.log.gz 2024-04-25T04:46:04.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124294.log 2024-04-25T04:46:04.288 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68937.log.gz 2024-04-25T04:46:04.289 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109510.log 2024-04-25T04:46:04.289 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124294.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124294.log.gz 2024-04-25T04:46:04.289 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151726.log 2024-04-25T04:46:04.290 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109510.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109510.log.gz 2024-04-25T04:46:04.290 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132595.log 2024-04-25T04:46:04.291 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151726.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151726.log.gz 2024-04-25T04:46:04.291 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86930.log 2024-04-25T04:46:04.292 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132595.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132595.log.gz 2024-04-25T04:46:04.292 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86293.log 2024-04-25T04:46:04.293 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86930.log.gz 2024-04-25T04:46:04.293 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83801.log 2024-04-25T04:46:04.293 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86293.log.gz 2024-04-25T04:46:04.294 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.99537.log 2024-04-25T04:46:04.294 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83801.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83801.log.gz 2024-04-25T04:46:04.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91413.log 2024-04-25T04:46:04.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.99537.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.99537.log.gz 2024-04-25T04:46:04.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54597.log 2024-04-25T04:46:04.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91413.log.gz 2024-04-25T04:46:04.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141717.log 2024-04-25T04:46:04.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54597.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54597.log.gz 2024-04-25T04:46:04.297 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142775.log 2024-04-25T04:46:04.298 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141717.log.gz 2024-04-25T04:46:04.298 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107540.log 2024-04-25T04:46:04.299 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142775.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142775.log.gz 2024-04-25T04:46:04.299 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88392.log 2024-04-25T04:46:04.300 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107540.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107540.log.gz 2024-04-25T04:46:04.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147242.log 2024-04-25T04:46:04.301 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88392.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88392.log.gz 2024-04-25T04:46:04.301 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138123.log 2024-04-25T04:46:04.302 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147242.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147242.log.gz 2024-04-25T04:46:04.302 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95772.log 2024-04-25T04:46:04.303 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138123.log.gz 2024-04-25T04:46:04.303 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63294.log 2024-04-25T04:46:04.303 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95772.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95772.log.gz 2024-04-25T04:46:04.304 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72831.log 2024-04-25T04:46:04.304 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63294.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63294.log.gz 2024-04-25T04:46:04.305 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104861.log 2024-04-25T04:46:04.305 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72831.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72831.log.gz 2024-04-25T04:46:04.306 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72747.log 2024-04-25T04:46:04.306 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104861.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104861.log.gz 2024-04-25T04:46:04.307 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43544.log 2024-04-25T04:46:04.308 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72747.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72747.log.gz 2024-04-25T04:46:04.308 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42650.log 2024-04-25T04:46:04.308 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43544.log.gz 2024-04-25T04:46:04.309 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85273.log 2024-04-25T04:46:04.309 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42650.log.gz 2024-04-25T04:46:04.310 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94630.log 2024-04-25T04:46:04.310 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85273.log.gz 2024-04-25T04:46:04.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128634.log 2024-04-25T04:46:04.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94630.log.gz 2024-04-25T04:46:04.312 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70236.log 2024-04-25T04:46:04.312 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128634.log.gz 2024-04-25T04:46:04.313 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153813.log 2024-04-25T04:46:04.313 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70236.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70236.log.gz 2024-04-25T04:46:04.314 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101644.log 2024-04-25T04:46:04.314 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153813.log.gz 2024-04-25T04:46:04.314 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44040.log 2024-04-25T04:46:04.315 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101644.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101644.log.gz 2024-04-25T04:46:04.316 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76858.log 2024-04-25T04:46:04.316 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44040.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44040.log.gz 2024-04-25T04:46:04.316 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41643.log 2024-04-25T04:46:04.317 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76858.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76858.log.gz 2024-04-25T04:46:04.317 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85464.log 2024-04-25T04:46:04.318 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.41643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41643.log.gz 2024-04-25T04:46:04.318 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115110.log 2024-04-25T04:46:04.319 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85464.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85464.log.gz 2024-04-25T04:46:04.319 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69373.log 2024-04-25T04:46:04.320 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.115110.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115110.log.gz 2024-04-25T04:46:04.320 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46385.log 2024-04-25T04:46:04.321 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69373.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69373.log.gz 2024-04-25T04:46:04.321 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102279.log 2024-04-25T04:46:04.322 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46385.log.gz 2024-04-25T04:46:04.322 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106819.log 2024-04-25T04:46:04.323 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102279.log.gz 2024-04-25T04:46:04.323 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156281.log 2024-04-25T04:46:04.324 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106819.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106819.log.gz 2024-04-25T04:46:04.324 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153262.log 2024-04-25T04:46:04.325 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156281.log.gz 2024-04-25T04:46:04.325 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71353.log 2024-04-25T04:46:04.326 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153262.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153262.log.gz 2024-04-25T04:46:04.326 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131128.log 2024-04-25T04:46:04.327 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71353.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71353.log.gz 2024-04-25T04:46:04.327 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101100.log 2024-04-25T04:46:04.327 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131128.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131128.log.gz 2024-04-25T04:46:04.328 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126658.log 2024-04-25T04:46:04.328 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101100.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101100.log.gz 2024-04-25T04:46:04.329 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84770.log 2024-04-25T04:46:04.329 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126658.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126658.log.gz 2024-04-25T04:46:04.330 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115857.log 2024-04-25T04:46:04.331 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84770.log.gz 2024-04-25T04:46:04.331 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70258.log 2024-04-25T04:46:04.331 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115857.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115857.log.gz 2024-04-25T04:46:04.332 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77258.log 2024-04-25T04:46:04.333 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70258.log.gz 2024-04-25T04:46:04.333 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101690.log 2024-04-25T04:46:04.333 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77258.log.gz 2024-04-25T04:46:04.334 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88984.log 2024-04-25T04:46:04.335 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101690.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101690.log.gz 2024-04-25T04:46:04.335 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149716.log 2024-04-25T04:46:04.336 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.88984.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88984.log.gz 2024-04-25T04:46:04.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74911.log 2024-04-25T04:46:04.336 INFO:teuthology.orchestra.run.smithi173.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-04-25T04:46:04.337 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149716.log.gz 2024-04-25T04:46:04.337 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46997.log 2024-04-25T04:46:04.337 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74911.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74911.log.gz 2024-04-25T04:46:04.338 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116852.log 2024-04-25T04:46:04.338 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46997.log.gz 2024-04-25T04:46:04.339 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82350.log 2024-04-25T04:46:04.339 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116852.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116852.log.gz 2024-04-25T04:46:04.339 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143788.log 2024-04-25T04:46:04.340 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82350.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82350.log.gz 2024-04-25T04:46:04.340 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102037.log 2024-04-25T04:46:04.341 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143788.log.gz 2024-04-25T04:46:04.341 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152302.log 2024-04-25T04:46:04.342 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102037.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102037.log.gz 2024-04-25T04:46:04.342 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98348.log 2024-04-25T04:46:04.343 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152302.log.gz 2024-04-25T04:46:04.343 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132279.log 2024-04-25T04:46:04.344 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98348.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98348.log.gz 2024-04-25T04:46:04.344 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46834.log 2024-04-25T04:46:04.345 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132279.log.gz 2024-04-25T04:46:04.345 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56895.log 2024-04-25T04:46:04.346 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46834.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46834.log.gz 2024-04-25T04:46:04.346 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49586.log 2024-04-25T04:46:04.347 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56895.log.gz 2024-04-25T04:46:04.347 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71435.log 2024-04-25T04:46:04.348 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49586.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49586.log.gz 2024-04-25T04:46:04.348 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140436.log 2024-04-25T04:46:04.349 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71435.log.gz 2024-04-25T04:46:04.349 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155187.log 2024-04-25T04:46:04.350 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140436.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140436.log.gz 2024-04-25T04:46:04.350 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57623.log 2024-04-25T04:46:04.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155187.log.gz 2024-04-25T04:46:04.351 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49458.log 2024-04-25T04:46:04.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57623.log.gz 2024-04-25T04:46:04.352 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70930.log 2024-04-25T04:46:04.352 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49458.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49458.log.gz 2024-04-25T04:46:04.352 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103721.log 2024-04-25T04:46:04.353 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70930.log.gz 2024-04-25T04:46:04.353 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154933.log 2024-04-25T04:46:04.354 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103721.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103721.log.gz 2024-04-25T04:46:04.354 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115752.log 2024-04-25T04:46:04.355 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154933.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154933.log.gz 2024-04-25T04:46:04.355 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139013.log 2024-04-25T04:46:04.356 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115752.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115752.log.gz 2024-04-25T04:46:04.356 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139727.log 2024-04-25T04:46:04.357 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139013.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139013.log.gz 2024-04-25T04:46:04.357 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43431.log 2024-04-25T04:46:04.358 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139727.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139727.log.gz 2024-04-25T04:46:04.358 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67634.log 2024-04-25T04:46:04.358 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43431.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43431.log.gz 2024-04-25T04:46:04.359 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146682.log 2024-04-25T04:46:04.360 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67634.log.gz 2024-04-25T04:46:04.360 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58500.log 2024-04-25T04:46:04.361 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146682.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146682.log.gz 2024-04-25T04:46:04.361 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83612.log 2024-04-25T04:46:04.362 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58500.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58500.log.gz 2024-04-25T04:46:04.362 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132763.log 2024-04-25T04:46:04.362 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83612.log.gz 2024-04-25T04:46:04.363 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56081.log 2024-04-25T04:46:04.363 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132763.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132763.log.gz 2024-04-25T04:46:04.364 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121327.log 2024-04-25T04:46:04.364 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56081.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56081.log.gz 2024-04-25T04:46:04.365 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59551.log 2024-04-25T04:46:04.365 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121327.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121327.log.gz 2024-04-25T04:46:04.365 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46826.log 2024-04-25T04:46:04.366 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59551.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59551.log.gz 2024-04-25T04:46:04.366 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106562.log 2024-04-25T04:46:04.367 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46826.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46826.log.gz 2024-04-25T04:46:04.367 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118322.log 2024-04-25T04:46:04.368 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106562.log.gz 2024-04-25T04:46:04.368 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47057.log 2024-04-25T04:46:04.369 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118322.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118322.log.gz 2024-04-25T04:46:04.369 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55743.log 2024-04-25T04:46:04.370 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47057.log.gz 2024-04-25T04:46:04.370 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131757.log 2024-04-25T04:46:04.371 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.55743.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55743.log.gz 2024-04-25T04:46:04.371 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109134.log 2024-04-25T04:46:04.371 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131757.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131757.log.gz 2024-04-25T04:46:04.372 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56082.log 2024-04-25T04:46:04.372 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109134.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109134.log.gz 2024-04-25T04:46:04.373 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124095.log 2024-04-25T04:46:04.373 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56082.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56082.log.gz 2024-04-25T04:46:04.373 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146205.log 2024-04-25T04:46:04.374 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124095.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124095.log.gz 2024-04-25T04:46:04.374 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58293.log 2024-04-25T04:46:04.375 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146205.log.gz 2024-04-25T04:46:04.375 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130778.log 2024-04-25T04:46:04.376 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58293.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58293.log.gz 2024-04-25T04:46:04.376 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144563.log 2024-04-25T04:46:04.376 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130778.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130778.log.gz 2024-04-25T04:46:04.377 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139937.log 2024-04-25T04:46:04.377 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144563.log.gz 2024-04-25T04:46:04.378 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57787.log 2024-04-25T04:46:04.378 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139937.log.gz 2024-04-25T04:46:04.378 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76045.log 2024-04-25T04:46:04.379 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57787.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57787.log.gz 2024-04-25T04:46:04.379 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118534.log 2024-04-25T04:46:04.380 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.76045.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76045.log.gz 2024-04-25T04:46:04.380 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89843.log 2024-04-25T04:46:04.381 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118534.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118534.log.gz 2024-04-25T04:46:04.381 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127394.log 2024-04-25T04:46:04.382 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89843.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89843.log.gz 2024-04-25T04:46:04.382 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159592.log 2024-04-25T04:46:04.383 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127394.log.gz 2024-04-25T04:46:04.383 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145676.log 2024-04-25T04:46:04.383 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159592.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159592.log.gz 2024-04-25T04:46:04.384 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105851.log 2024-04-25T04:46:04.384 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145676.log.gz 2024-04-25T04:46:04.385 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53897.log 2024-04-25T04:46:04.385 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105851.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105851.log.gz 2024-04-25T04:46:04.385 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48764.log 2024-04-25T04:46:04.386 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.53897.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53897.log.gz 2024-04-25T04:46:04.386 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74933.log 2024-04-25T04:46:04.387 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48764.log.gz 2024-04-25T04:46:04.387 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159423.log 2024-04-25T04:46:04.388 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74933.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74933.log.gz 2024-04-25T04:46:04.388 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121043.log 2024-04-25T04:46:04.389 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159423.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159423.log.gz 2024-04-25T04:46:04.389 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143341.log 2024-04-25T04:46:04.389 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121043.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121043.log.gz 2024-04-25T04:46:04.390 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156491.log 2024-04-25T04:46:04.390 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143341.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143341.log.gz 2024-04-25T04:46:04.391 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153770.log 2024-04-25T04:46:04.391 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156491.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156491.log.gz 2024-04-25T04:46:04.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138908.log 2024-04-25T04:46:04.392 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153770.log.gz 2024-04-25T04:46:04.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144542.log 2024-04-25T04:46:04.393 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138908.log.gz 2024-04-25T04:46:04.393 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152869.log 2024-04-25T04:46:04.394 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144542.log.gz 2024-04-25T04:46:04.394 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138518.log 2024-04-25T04:46:04.395 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152869.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152869.log.gz 2024-04-25T04:46:04.395 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150202.log 2024-04-25T04:46:04.396 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138518.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138518.log.gz 2024-04-25T04:46:04.396 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159471.log 2024-04-25T04:46:04.397 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150202.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150202.log.gz 2024-04-25T04:46:04.397 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89729.log 2024-04-25T04:46:04.397 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159471.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159471.log.gz 2024-04-25T04:46:04.398 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151704.log 2024-04-25T04:46:04.398 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89729.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89729.log.gz 2024-04-25T04:46:04.399 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52129.log 2024-04-25T04:46:04.399 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151704.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151704.log.gz 2024-04-25T04:46:04.399 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103302.log 2024-04-25T04:46:04.400 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52129.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52129.log.gz 2024-04-25T04:46:04.400 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144124.log 2024-04-25T04:46:04.401 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103302.log.gz 2024-04-25T04:46:04.401 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58042.log 2024-04-25T04:46:04.402 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144124.log.gz 2024-04-25T04:46:04.402 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132067.log 2024-04-25T04:46:04.402 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58042.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58042.log.gz 2024-04-25T04:46:04.403 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70321.log 2024-04-25T04:46:04.403 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132067.log.gz 2024-04-25T04:46:04.404 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68928.log 2024-04-25T04:46:04.404 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70321.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70321.log.gz 2024-04-25T04:46:04.405 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138865.log 2024-04-25T04:46:04.405 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68928.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68928.log.gz 2024-04-25T04:46:04.405 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132450.log 2024-04-25T04:46:04.406 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138865.log.gz 2024-04-25T04:46:04.406 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155060.log 2024-04-25T04:46:04.407 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132450.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132450.log.gz 2024-04-25T04:46:04.407 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94442.log 2024-04-25T04:46:04.408 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155060.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155060.log.gz 2024-04-25T04:46:04.408 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70137.log 2024-04-25T04:46:04.409 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94442.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94442.log.gz 2024-04-25T04:46:04.409 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79543.log 2024-04-25T04:46:04.410 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70137.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70137.log.gz 2024-04-25T04:46:04.410 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65207.log 2024-04-25T04:46:04.410 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79543.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79543.log.gz 2024-04-25T04:46:04.411 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42516.log 2024-04-25T04:46:04.411 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65207.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65207.log.gz 2024-04-25T04:46:04.411 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141043.log 2024-04-25T04:46:04.412 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42516.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42516.log.gz 2024-04-25T04:46:04.412 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89615.log 2024-04-25T04:46:04.413 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141043.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141043.log.gz 2024-04-25T04:46:04.413 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102145.log 2024-04-25T04:46:04.414 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89615.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89615.log.gz 2024-04-25T04:46:04.414 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139988.log 2024-04-25T04:46:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102145.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102145.log.gz 2024-04-25T04:46:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108608.log 2024-04-25T04:46:04.415 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139988.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139988.log.gz 2024-04-25T04:46:04.416 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78894.log 2024-04-25T04:46:04.417 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108608.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108608.log.gz 2024-04-25T04:46:04.417 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142903.log 2024-04-25T04:46:04.417 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78894.log.gz 2024-04-25T04:46:04.417 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79903.log 2024-04-25T04:46:04.418 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142903.log.gz 2024-04-25T04:46:04.418 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101597.log 2024-04-25T04:46:04.419 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79903.log.gz 2024-04-25T04:46:04.419 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122266.log 2024-04-25T04:46:04.420 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101597.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101597.log.gz 2024-04-25T04:46:04.420 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103511.log 2024-04-25T04:46:04.420 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122266.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122266.log.gz 2024-04-25T04:46:04.421 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97197.log 2024-04-25T04:46:04.422 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103511.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103511.log.gz 2024-04-25T04:46:04.422 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149540.log 2024-04-25T04:46:04.422 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97197.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97197.log.gz 2024-04-25T04:46:04.422 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58629.log 2024-04-25T04:46:04.423 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149540.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149540.log.gz 2024-04-25T04:46:04.423 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116298.log 2024-04-25T04:46:04.424 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58629.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58629.log.gz 2024-04-25T04:46:04.424 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69355.log 2024-04-25T04:46:04.425 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116298.log.gz 2024-04-25T04:46:04.425 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51246.log 2024-04-25T04:46:04.426 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69355.log.gz 2024-04-25T04:46:04.426 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71822.log 2024-04-25T04:46:04.427 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51246.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51246.log.gz 2024-04-25T04:46:04.427 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154598.log 2024-04-25T04:46:04.427 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71822.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71822.log.gz 2024-04-25T04:46:04.427 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98990.log 2024-04-25T04:46:04.428 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154598.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154598.log.gz 2024-04-25T04:46:04.428 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116275.log 2024-04-25T04:46:04.429 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98990.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98990.log.gz 2024-04-25T04:46:04.429 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61638.log 2024-04-25T04:46:04.430 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116275.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116275.log.gz 2024-04-25T04:46:04.430 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155398.log 2024-04-25T04:46:04.431 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61638.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61638.log.gz 2024-04-25T04:46:04.431 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95900.log 2024-04-25T04:46:04.432 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155398.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155398.log.gz 2024-04-25T04:46:04.432 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95155.log 2024-04-25T04:46:04.432 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95900.log.gz 2024-04-25T04:46:04.433 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145989.log 2024-04-25T04:46:04.433 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95155.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95155.log.gz 2024-04-25T04:46:04.433 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133177.log 2024-04-25T04:46:04.434 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145989.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145989.log.gz 2024-04-25T04:46:04.434 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105139.log 2024-04-25T04:46:04.435 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133177.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133177.log.gz 2024-04-25T04:46:04.435 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56457.log 2024-04-25T04:46:04.436 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105139.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105139.log.gz 2024-04-25T04:46:04.436 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70534.log 2024-04-25T04:46:04.437 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56457.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56457.log.gz 2024-04-25T04:46:04.437 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157251.log 2024-04-25T04:46:04.438 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70534.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70534.log.gz 2024-04-25T04:46:04.438 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83804.log 2024-04-25T04:46:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157251.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157251.log.gz 2024-04-25T04:46:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44164.log 2024-04-25T04:46:04.439 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83804.log.gz 2024-04-25T04:46:04.440 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109535.log 2024-04-25T04:46:04.441 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44164.log.gz 2024-04-25T04:46:04.441 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93686.log 2024-04-25T04:46:04.441 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109535.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109535.log.gz 2024-04-25T04:46:04.442 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40458.log 2024-04-25T04:46:04.442 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93686.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93686.log.gz 2024-04-25T04:46:04.442 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98459.log 2024-04-25T04:46:04.443 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40458.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40458.log.gz 2024-04-25T04:46:04.443 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66905.log 2024-04-25T04:46:04.444 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98459.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98459.log.gz 2024-04-25T04:46:04.444 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149180.log 2024-04-25T04:46:04.445 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66905.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66905.log.gz 2024-04-25T04:46:04.445 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93265.log 2024-04-25T04:46:04.446 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149180.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149180.log.gz 2024-04-25T04:46:04.446 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160432.log 2024-04-25T04:46:04.446 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93265.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93265.log.gz 2024-04-25T04:46:04.447 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119442.log 2024-04-25T04:46:04.448 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160432.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160432.log.gz 2024-04-25T04:46:04.448 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151495.log 2024-04-25T04:46:04.448 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119442.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119442.log.gz 2024-04-25T04:46:04.449 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150404.log 2024-04-25T04:46:04.449 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151495.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151495.log.gz 2024-04-25T04:46:04.449 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141991.log 2024-04-25T04:46:04.450 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150404.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150404.log.gz 2024-04-25T04:46:04.450 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55308.log 2024-04-25T04:46:04.451 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141991.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141991.log.gz 2024-04-25T04:46:04.451 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120208.log 2024-04-25T04:46:04.452 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55308.log.gz 2024-04-25T04:46:04.452 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155143.log 2024-04-25T04:46:04.453 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120208.log.gz 2024-04-25T04:46:04.453 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119151.log 2024-04-25T04:46:04.454 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155143.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155143.log.gz 2024-04-25T04:46:04.454 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52245.log 2024-04-25T04:46:04.454 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119151.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119151.log.gz 2024-04-25T04:46:04.455 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49971.log 2024-04-25T04:46:04.455 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.52245.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52245.log.gz 2024-04-25T04:46:04.455 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59131.log 2024-04-25T04:46:04.456 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49971.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49971.log.gz 2024-04-25T04:46:04.456 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70135.log 2024-04-25T04:46:04.457 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59131.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59131.log.gz 2024-04-25T04:46:04.457 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149046.log 2024-04-25T04:46:04.458 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70135.log.gz 2024-04-25T04:46:04.458 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71059.log 2024-04-25T04:46:04.459 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149046.log.gz 2024-04-25T04:46:04.459 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137590.log 2024-04-25T04:46:04.459 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71059.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71059.log.gz 2024-04-25T04:46:04.460 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62788.log 2024-04-25T04:46:04.460 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137590.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137590.log.gz 2024-04-25T04:46:04.461 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108797.log 2024-04-25T04:46:04.461 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62788.log.gz 2024-04-25T04:46:04.462 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160970.log 2024-04-25T04:46:04.462 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108797.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108797.log.gz 2024-04-25T04:46:04.463 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103303.log 2024-04-25T04:46:04.463 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.160970.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160970.log.gz 2024-04-25T04:46:04.463 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57193.log 2024-04-25T04:46:04.464 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103303.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103303.log.gz 2024-04-25T04:46:04.464 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104141.log 2024-04-25T04:46:04.465 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57193.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57193.log.gz 2024-04-25T04:46:04.465 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114514.log 2024-04-25T04:46:04.466 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104141.log.gz 2024-04-25T04:46:04.466 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69863.log 2024-04-25T04:46:04.467 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114514.log.gz 2024-04-25T04:46:04.467 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158296.log 2024-04-25T04:46:04.467 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69863.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69863.log.gz 2024-04-25T04:46:04.468 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121378.log 2024-04-25T04:46:04.468 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158296.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158296.log.gz 2024-04-25T04:46:04.469 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143955.log 2024-04-25T04:46:04.469 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121378.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121378.log.gz 2024-04-25T04:46:04.469 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101903.log 2024-04-25T04:46:04.470 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143955.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143955.log.gz 2024-04-25T04:46:04.470 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158371.log 2024-04-25T04:46:04.471 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101903.log.gz 2024-04-25T04:46:04.471 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140248.log 2024-04-25T04:46:04.472 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.158371.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158371.log.gz 2024-04-25T04:46:04.472 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120769.log 2024-04-25T04:46:04.473 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140248.log.gz 2024-04-25T04:46:04.473 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138031.log 2024-04-25T04:46:04.474 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120769.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120769.log.gz 2024-04-25T04:46:04.474 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81569.log 2024-04-25T04:46:04.474 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138031.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138031.log.gz 2024-04-25T04:46:04.475 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145830.log 2024-04-25T04:46:04.475 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81569.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81569.log.gz 2024-04-25T04:46:04.475 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-04-25T04:46:04.476 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145830.log.gz 2024-04-25T04:46:04.476 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157527.log 2024-04-25T04:46:04.477 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148162.log 2024-04-25T04:46:04.478 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157527.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157527.log.gz 2024-04-25T04:46:04.478 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72072.log 2024-04-25T04:46:04.478 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148162.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148162.log.gz 2024-04-25T04:46:04.479 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131650.log 2024-04-25T04:46:04.479 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72072.log.gz 2024-04-25T04:46:04.480 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57935.log 2024-04-25T04:46:04.480 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131650.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131650.log.gz 2024-04-25T04:46:04.481 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86292.log 2024-04-25T04:46:04.481 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57935.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57935.log.gz 2024-04-25T04:46:04.481 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93433.log 2024-04-25T04:46:04.482 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86292.log.gz 2024-04-25T04:46:04.482 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129329.log 2024-04-25T04:46:04.483 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93433.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93433.log.gz 2024-04-25T04:46:04.483 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138804.log 2024-04-25T04:46:04.484 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129329.log.gz 2024-04-25T04:46:04.484 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130530.log 2024-04-25T04:46:04.485 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138804.log.gz 2024-04-25T04:46:04.485 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102550.log 2024-04-25T04:46:04.485 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130530.log.gz 2024-04-25T04:46:04.486 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86709.log 2024-04-25T04:46:04.486 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102550.log.gz 2024-04-25T04:46:04.487 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122486.log 2024-04-25T04:46:04.487 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86709.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86709.log.gz 2024-04-25T04:46:04.487 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70678.log 2024-04-25T04:46:04.488 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122486.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122486.log.gz 2024-04-25T04:46:04.488 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101234.log 2024-04-25T04:46:04.489 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70678.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70678.log.gz 2024-04-25T04:46:04.489 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71438.log 2024-04-25T04:46:04.490 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101234.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101234.log.gz 2024-04-25T04:46:04.490 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142864.log 2024-04-25T04:46:04.491 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71438.log.gz 2024-04-25T04:46:04.491 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68198.log 2024-04-25T04:46:04.491 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142864.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142864.log.gz 2024-04-25T04:46:04.492 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130529.log 2024-04-25T04:46:04.492 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68198.log.gz 2024-04-25T04:46:04.493 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120095.log 2024-04-25T04:46:04.493 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130529.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130529.log.gz 2024-04-25T04:46:04.493 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155229.log 2024-04-25T04:46:04.494 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120095.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120095.log.gz 2024-04-25T04:46:04.494 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63533.log 2024-04-25T04:46:04.495 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155229.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155229.log.gz 2024-04-25T04:46:04.495 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58562.log 2024-04-25T04:46:04.496 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.63533.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63533.log.gz 2024-04-25T04:46:04.496 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53876.log 2024-04-25T04:46:04.497 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58562.log.gz 2024-04-25T04:46:04.497 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96003.log 2024-04-25T04:46:04.498 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53876.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53876.log.gz 2024-04-25T04:46:04.498 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157442.log 2024-04-25T04:46:04.499 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96003.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96003.log.gz 2024-04-25T04:46:04.499 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.110910.log 2024-04-25T04:46:04.500 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157442.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157442.log.gz 2024-04-25T04:46:04.500 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62459.log 2024-04-25T04:46:04.501 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.110910.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.110910.log.gz 2024-04-25T04:46:04.501 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123822.log 2024-04-25T04:46:04.502 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62459.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62459.log.gz 2024-04-25T04:46:04.503 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45943.log 2024-04-25T04:46:04.503 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107203.log 2024-04-25T04:46:04.504 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123822.log.gz 2024-04-25T04:46:04.504 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45943.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45943.log.gz 2024-04-25T04:46:04.505 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114033.log 2024-04-25T04:46:04.505 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107203.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107203.log.gz 2024-04-25T04:46:04.505 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69837.log 2024-04-25T04:46:04.506 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114033.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114033.log.gz 2024-04-25T04:46:04.506 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78413.log 2024-04-25T04:46:04.507 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69837.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69837.log.gz 2024-04-25T04:46:04.507 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62321.log 2024-04-25T04:46:04.508 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78413.log.gz 2024-04-25T04:46:04.508 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83396.log 2024-04-25T04:46:04.509 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62321.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62321.log.gz 2024-04-25T04:46:04.509 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87509.log 2024-04-25T04:46:04.510 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83396.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83396.log.gz 2024-04-25T04:46:04.510 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58670.log 2024-04-25T04:46:04.511 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87509.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87509.log.gz 2024-04-25T04:46:04.511 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133996.log 2024-04-25T04:46:04.511 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58670.log.gz 2024-04-25T04:46:04.512 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96697.log 2024-04-25T04:46:04.512 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133996.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133996.log.gz 2024-04-25T04:46:04.513 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132237.log 2024-04-25T04:46:04.513 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96697.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96697.log.gz 2024-04-25T04:46:04.514 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125747.log 2024-04-25T04:46:04.514 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132237.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132237.log.gz 2024-04-25T04:46:04.515 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127286.log 2024-04-25T04:46:04.515 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125747.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125747.log.gz 2024-04-25T04:46:04.516 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81230.log 2024-04-25T04:46:04.516 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127286.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127286.log.gz 2024-04-25T04:46:04.516 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161460.log 2024-04-25T04:46:04.517 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81230.log.gz 2024-04-25T04:46:04.518 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108946.log 2024-04-25T04:46:04.518 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161460.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161460.log.gz 2024-04-25T04:46:04.518 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58164.log 2024-04-25T04:46:04.519 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108946.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108946.log.gz 2024-04-25T04:46:04.519 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-04-25T04:46:04.520 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58164.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58164.log.gz 2024-04-25T04:46:04.520 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94672.log 2024-04-25T04:46:04.521 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54658.log 2024-04-25T04:46:04.522 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94672.log.gz 2024-04-25T04:46:04.522 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118742.log 2024-04-25T04:46:04.523 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54658.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54658.log.gz 2024-04-25T04:46:04.523 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77903.log 2024-04-25T04:46:04.523 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118742.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118742.log.gz 2024-04-25T04:46:04.524 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84937.log 2024-04-25T04:46:04.524 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.77903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77903.log.gz 2024-04-25T04:46:04.525 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124324.log 2024-04-25T04:46:04.525 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84937.log.gz 2024-04-25T04:46:04.526 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159255.log 2024-04-25T04:46:04.526 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124324.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124324.log.gz 2024-04-25T04:46:04.527 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109370.log 2024-04-25T04:46:04.527 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159255.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159255.log.gz 2024-04-25T04:46:04.528 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73500.log 2024-04-25T04:46:04.528 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109370.log.gz 2024-04-25T04:46:04.529 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63402.log 2024-04-25T04:46:04.529 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73500.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73500.log.gz 2024-04-25T04:46:04.530 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77076.log 2024-04-25T04:46:04.530 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63402.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63402.log.gz 2024-04-25T04:46:04.531 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141781.log 2024-04-25T04:46:04.531 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77076.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77076.log.gz 2024-04-25T04:46:04.532 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71780.log 2024-04-25T04:46:04.533 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141781.log.gz 2024-04-25T04:46:04.533 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152645.log 2024-04-25T04:46:04.533 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71780.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71780.log.gz 2024-04-25T04:46:04.533 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116810.log 2024-04-25T04:46:04.534 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152645.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152645.log.gz 2024-04-25T04:46:04.534 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65603.log 2024-04-25T04:46:04.535 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116810.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116810.log.gz 2024-04-25T04:46:04.535 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157441.log 2024-04-25T04:46:04.536 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65603.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65603.log.gz 2024-04-25T04:46:04.536 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85699.log 2024-04-25T04:46:04.537 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157441.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157441.log.gz 2024-04-25T04:46:04.537 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41909.log 2024-04-25T04:46:04.538 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85699.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85699.log.gz 2024-04-25T04:46:04.538 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71620.log 2024-04-25T04:46:04.540 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41909.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41909.log.gz 2024-04-25T04:46:04.540 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56918.log 2024-04-25T04:46:04.540 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71620.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71620.log.gz 2024-04-25T04:46:04.540 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116916.log 2024-04-25T04:46:04.541 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56918.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56918.log.gz 2024-04-25T04:46:04.541 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72073.log 2024-04-25T04:46:04.542 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116916.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116916.log.gz 2024-04-25T04:46:04.542 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103407.log 2024-04-25T04:46:04.543 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72073.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72073.log.gz 2024-04-25T04:46:04.543 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49375.log 2024-04-25T04:46:04.544 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103407.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103407.log.gz 2024-04-25T04:46:04.544 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141865.log 2024-04-25T04:46:04.545 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49375.log.gz 2024-04-25T04:46:04.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55189.log 2024-04-25T04:46:04.546 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141865.log.gz 2024-04-25T04:46:04.546 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61238.log 2024-04-25T04:46:04.547 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55189.log.gz 2024-04-25T04:46:04.547 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89562.log 2024-04-25T04:46:04.550 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61238.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61238.log.gz 2024-04-25T04:46:04.550 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144846.log 2024-04-25T04:46:04.550 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89562.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89562.log.gz 2024-04-25T04:46:04.550 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148445.log 2024-04-25T04:46:04.550 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144846.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144846.log.gz 2024-04-25T04:46:04.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138406.log 2024-04-25T04:46:04.551 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89817.log 2024-04-25T04:46:04.551 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148445.log.gz 2024-04-25T04:46:04.552 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.138406.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138406.log.gz 2024-04-25T04:46:04.552 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120283.log 2024-04-25T04:46:04.553 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89817.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89817.log.gz 2024-04-25T04:46:04.553 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40507.log 2024-04-25T04:46:04.554 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120283.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120283.log.gz 2024-04-25T04:46:04.555 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119316.log 2024-04-25T04:46:04.556 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40507.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40507.log.gz 2024-04-25T04:46:04.556 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81778.log 2024-04-25T04:46:04.557 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119316.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119316.log.gz 2024-04-25T04:46:04.557 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99160.log 2024-04-25T04:46:04.557 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81778.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81778.log.gz 2024-04-25T04:46:04.558 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59682.log 2024-04-25T04:46:04.558 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99160.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99160.log.gz 2024-04-25T04:46:04.558 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118533.log 2024-04-25T04:46:04.559 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59682.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59682.log.gz 2024-04-25T04:46:04.559 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159994.log 2024-04-25T04:46:04.560 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118533.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118533.log.gz 2024-04-25T04:46:04.560 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95920.log 2024-04-25T04:46:04.561 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.159994.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159994.log.gz 2024-04-25T04:46:04.561 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52848.log 2024-04-25T04:46:04.562 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95920.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95920.log.gz 2024-04-25T04:46:04.562 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103364.log 2024-04-25T04:46:04.563 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52848.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52848.log.gz 2024-04-25T04:46:04.563 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130507.log 2024-04-25T04:46:04.564 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103364.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103364.log.gz 2024-04-25T04:46:04.564 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43367.log 2024-04-25T04:46:04.565 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130507.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130507.log.gz 2024-04-25T04:46:04.565 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152386.log 2024-04-25T04:46:04.566 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43367.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43367.log.gz 2024-04-25T04:46:04.566 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103468.log 2024-04-25T04:46:04.567 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152386.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152386.log.gz 2024-04-25T04:46:04.567 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158100.log 2024-04-25T04:46:04.568 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103468.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103468.log.gz 2024-04-25T04:46:04.568 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109336.log 2024-04-25T04:46:04.569 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158100.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158100.log.gz 2024-04-25T04:46:04.569 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126833.log 2024-04-25T04:46:04.570 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109336.log.gz 2024-04-25T04:46:04.570 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153057.log 2024-04-25T04:46:04.570 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126833.log.gz 2024-04-25T04:46:04.571 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85003.log 2024-04-25T04:46:04.572 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153057.log.gz 2024-04-25T04:46:04.572 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90634.log 2024-04-25T04:46:04.572 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85003.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85003.log.gz 2024-04-25T04:46:04.572 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88943.log 2024-04-25T04:46:04.573 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90634.log.gz 2024-04-25T04:46:04.573 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75746.log 2024-04-25T04:46:04.574 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88943.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88943.log.gz 2024-04-25T04:46:04.574 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155020.log 2024-04-25T04:46:04.575 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75746.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75746.log.gz 2024-04-25T04:46:04.575 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67700.log 2024-04-25T04:46:04.576 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155020.log.gz 2024-04-25T04:46:04.576 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117252.log 2024-04-25T04:46:04.576 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67700.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67700.log.gz 2024-04-25T04:46:04.577 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135779.log 2024-04-25T04:46:04.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117252.log.gz 2024-04-25T04:46:04.578 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112163.log 2024-04-25T04:46:04.578 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135779.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135779.log.gz 2024-04-25T04:46:04.579 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134384.log 2024-04-25T04:46:04.580 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112163.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112163.log.gz 2024-04-25T04:46:04.580 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141023.log 2024-04-25T04:46:04.581 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134384.log.gz 2024-04-25T04:46:04.581 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72411.log 2024-04-25T04:46:04.581 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141023.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141023.log.gz 2024-04-25T04:46:04.582 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56205.log 2024-04-25T04:46:04.583 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72411.log.gz 2024-04-25T04:46:04.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147598.log 2024-04-25T04:46:04.583 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56205.log.gz 2024-04-25T04:46:04.584 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124838.log 2024-04-25T04:46:04.585 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147598.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147598.log.gz 2024-04-25T04:46:04.585 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60881.log 2024-04-25T04:46:04.586 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92565.log 2024-04-25T04:46:04.586 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124838.log.gz 2024-04-25T04:46:04.586 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60881.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60881.log.gz 2024-04-25T04:46:04.587 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132176.log 2024-04-25T04:46:04.587 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92565.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92565.log.gz 2024-04-25T04:46:04.588 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121955.log 2024-04-25T04:46:04.588 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132176.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132176.log.gz 2024-04-25T04:46:04.589 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50057.log 2024-04-25T04:46:04.589 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121955.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121955.log.gz 2024-04-25T04:46:04.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78281.log 2024-04-25T04:46:04.590 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50057.log.gz 2024-04-25T04:46:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58460.log 2024-04-25T04:46:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78281.log.gz 2024-04-25T04:46:04.591 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85626.log 2024-04-25T04:46:04.592 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58460.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58460.log.gz 2024-04-25T04:46:04.593 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82825.log 2024-04-25T04:46:04.593 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85626.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85626.log.gz 2024-04-25T04:46:04.594 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117334.log 2024-04-25T04:46:04.594 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82825.log.gz 2024-04-25T04:46:04.595 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105219.log 2024-04-25T04:46:04.595 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117334.log.gz 2024-04-25T04:46:04.596 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130301.log 2024-04-25T04:46:04.597 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105219.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105219.log.gz 2024-04-25T04:46:04.597 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106901.log 2024-04-25T04:46:04.598 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130301.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130301.log.gz 2024-04-25T04:46:04.598 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120096.log 2024-04-25T04:46:04.599 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106901.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106901.log.gz 2024-04-25T04:46:04.599 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48387.log 2024-04-25T04:46:04.599 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120096.log.gz 2024-04-25T04:46:04.600 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105009.log 2024-04-25T04:46:04.601 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48387.log.gz 2024-04-25T04:46:04.601 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128380.log 2024-04-25T04:46:04.601 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105009.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105009.log.gz 2024-04-25T04:46:04.602 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76317.log 2024-04-25T04:46:04.603 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128380.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128380.log.gz 2024-04-25T04:46:04.603 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154361.log 2024-04-25T04:46:04.604 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76317.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76317.log.gz 2024-04-25T04:46:04.604 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123684.log 2024-04-25T04:46:04.604 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154361.log.gz 2024-04-25T04:46:04.605 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48245.log 2024-04-25T04:46:04.606 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123684.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123684.log.gz 2024-04-25T04:46:04.606 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124332.log 2024-04-25T04:46:04.606 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48245.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48245.log.gz 2024-04-25T04:46:04.607 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105345.log 2024-04-25T04:46:04.607 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124332.log.gz 2024-04-25T04:46:04.608 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145521.log 2024-04-25T04:46:04.608 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105345.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105345.log.gz 2024-04-25T04:46:04.608 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76811.log 2024-04-25T04:46:04.609 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145521.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145521.log.gz 2024-04-25T04:46:04.609 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110288.log 2024-04-25T04:46:04.610 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76811.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76811.log.gz 2024-04-25T04:46:04.610 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94905.log 2024-04-25T04:46:04.611 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110288.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110288.log.gz 2024-04-25T04:46:04.611 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152214.log 2024-04-25T04:46:04.612 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94905.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94905.log.gz 2024-04-25T04:46:04.612 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139182.log 2024-04-25T04:46:04.613 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152214.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152214.log.gz 2024-04-25T04:46:04.613 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117567.log 2024-04-25T04:46:04.614 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139182.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139182.log.gz 2024-04-25T04:46:04.614 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151788.log 2024-04-25T04:46:04.615 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117567.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117567.log.gz 2024-04-25T04:46:04.615 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154067.log 2024-04-25T04:46:04.616 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151788.log.gz 2024-04-25T04:46:04.616 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.102502.log 2024-04-25T04:46:04.617 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154067.log.gz 2024-04-25T04:46:04.617 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96530.log 2024-04-25T04:46:04.617 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.102502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.102502.log.gz 2024-04-25T04:46:04.618 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59198.log 2024-04-25T04:46:04.618 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96530.log.gz 2024-04-25T04:46:04.618 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95651.log 2024-04-25T04:46:04.619 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59198.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59198.log.gz 2024-04-25T04:46:04.619 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97533.log 2024-04-25T04:46:04.620 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95651.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95651.log.gz 2024-04-25T04:46:04.620 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66308.log 2024-04-25T04:46:04.621 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97533.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97533.log.gz 2024-04-25T04:46:04.621 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79963.log 2024-04-25T04:46:04.622 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66308.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66308.log.gz 2024-04-25T04:46:04.622 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101576.log 2024-04-25T04:46:04.623 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79963.log.gz 2024-04-25T04:46:04.623 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121655.log 2024-04-25T04:46:04.624 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101576.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101576.log.gz 2024-04-25T04:46:04.624 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75636.log 2024-04-25T04:46:04.625 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121655.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121655.log.gz 2024-04-25T04:46:04.625 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66996.log 2024-04-25T04:46:04.626 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75636.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75636.log.gz 2024-04-25T04:46:04.626 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89790.log 2024-04-25T04:46:04.627 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66996.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66996.log.gz 2024-04-25T04:46:04.627 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123600.log 2024-04-25T04:46:04.628 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89790.log.gz 2024-04-25T04:46:04.628 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137744.log 2024-04-25T04:46:04.629 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123600.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123600.log.gz 2024-04-25T04:46:04.629 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131230.log 2024-04-25T04:46:04.630 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137744.log.gz 2024-04-25T04:46:04.630 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144461.log 2024-04-25T04:46:04.631 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131230.log.gz 2024-04-25T04:46:04.631 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.87487.log 2024-04-25T04:46:04.632 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144461.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144461.log.gz 2024-04-25T04:46:04.632 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60020.log 2024-04-25T04:46:04.633 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.87487.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.87487.log.gz 2024-04-25T04:46:04.633 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138492.log 2024-04-25T04:46:04.634 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60020.log.gz 2024-04-25T04:46:04.634 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76818.log 2024-04-25T04:46:04.635 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138492.log.gz 2024-04-25T04:46:04.635 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47056.log 2024-04-25T04:46:04.636 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76818.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76818.log.gz 2024-04-25T04:46:04.636 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123936.log 2024-04-25T04:46:04.637 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47056.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47056.log.gz 2024-04-25T04:46:04.638 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82212.log 2024-04-25T04:46:04.638 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.123936.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.123936.log.gz 2024-04-25T04:46:04.638 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112430.log 2024-04-25T04:46:04.639 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82212.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82212.log.gz 2024-04-25T04:46:04.639 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116638.log 2024-04-25T04:46:04.640 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112430.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112430.log.gz 2024-04-25T04:46:04.640 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135572.log 2024-04-25T04:46:04.641 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116638.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116638.log.gz 2024-04-25T04:46:04.641 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.101416.log 2024-04-25T04:46:04.642 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135572.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135572.log.gz 2024-04-25T04:46:04.642 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42902.log 2024-04-25T04:46:04.643 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.101416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.101416.log.gz 2024-04-25T04:46:04.643 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47694.log 2024-04-25T04:46:04.644 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42902.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42902.log.gz 2024-04-25T04:46:04.644 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136786.log 2024-04-25T04:46:04.645 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47694.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47694.log.gz 2024-04-25T04:46:04.646 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158556.log 2024-04-25T04:46:04.646 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136786.log.gz 2024-04-25T04:46:04.647 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129711.log 2024-04-25T04:46:04.647 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158556.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158556.log.gz 2024-04-25T04:46:04.647 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127937.log 2024-04-25T04:46:04.648 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129711.log.gz 2024-04-25T04:46:04.648 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129538.log 2024-04-25T04:46:04.649 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127937.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127937.log.gz 2024-04-25T04:46:04.649 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144847.log 2024-04-25T04:46:04.650 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129538.log.gz 2024-04-25T04:46:04.650 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55794.log 2024-04-25T04:46:04.651 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144847.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144847.log.gz 2024-04-25T04:46:04.651 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66153.log 2024-04-25T04:46:04.652 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55794.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55794.log.gz 2024-04-25T04:46:04.652 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95374.log 2024-04-25T04:46:04.653 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.66153.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66153.log.gz 2024-04-25T04:46:04.653 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.158097.log 2024-04-25T04:46:04.654 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95374.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95374.log.gz 2024-04-25T04:46:04.654 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83319.log 2024-04-25T04:46:04.655 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.158097.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.158097.log.gz 2024-04-25T04:46:04.655 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82938.log 2024-04-25T04:46:04.656 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83319.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83319.log.gz 2024-04-25T04:46:04.656 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90012.log 2024-04-25T04:46:04.657 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82938.log.gz 2024-04-25T04:46:04.657 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84099.log 2024-04-25T04:46:04.658 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90012.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90012.log.gz 2024-04-25T04:46:04.658 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53204.log 2024-04-25T04:46:04.659 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84099.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84099.log.gz 2024-04-25T04:46:04.659 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150448.log 2024-04-25T04:46:04.660 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53204.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53204.log.gz 2024-04-25T04:46:04.660 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76119.log 2024-04-25T04:46:04.661 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150448.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150448.log.gz 2024-04-25T04:46:04.661 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78764.log 2024-04-25T04:46:04.662 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76119.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76119.log.gz 2024-04-25T04:46:04.662 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103123.log 2024-04-25T04:46:04.662 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78764.log.gz 2024-04-25T04:46:04.663 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79204.log 2024-04-25T04:46:04.664 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103123.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103123.log.gz 2024-04-25T04:46:04.664 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151392.log 2024-04-25T04:46:04.664 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79204.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79204.log.gz 2024-04-25T04:46:04.665 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43894.log 2024-04-25T04:46:04.665 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151392.log.gz 2024-04-25T04:46:04.666 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.54373.log 2024-04-25T04:46:04.666 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43894.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43894.log.gz 2024-04-25T04:46:04.667 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51150.log 2024-04-25T04:46:04.667 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.54373.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.54373.log.gz 2024-04-25T04:46:04.668 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.134587.log 2024-04-25T04:46:04.668 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51150.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51150.log.gz 2024-04-25T04:46:04.668 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110639.log 2024-04-25T04:46:04.669 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.134587.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.134587.log.gz 2024-04-25T04:46:04.669 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155965.log 2024-04-25T04:46:04.670 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110639.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110639.log.gz 2024-04-25T04:46:04.670 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117611.log 2024-04-25T04:46:04.671 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155965.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155965.log.gz 2024-04-25T04:46:04.671 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95290.log 2024-04-25T04:46:04.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117611.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117611.log.gz 2024-04-25T04:46:04.672 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105558.log 2024-04-25T04:46:04.672 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95290.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95290.log.gz 2024-04-25T04:46:04.673 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97600.log 2024-04-25T04:46:04.673 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105558.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105558.log.gz 2024-04-25T04:46:04.674 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115334.log 2024-04-25T04:46:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97600.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97600.log.gz 2024-04-25T04:46:04.675 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57788.log 2024-04-25T04:46:04.676 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115334.log.gz 2024-04-25T04:46:04.676 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152766.log 2024-04-25T04:46:04.677 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57788.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57788.log.gz 2024-04-25T04:46:04.677 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152636.log 2024-04-25T04:46:04.677 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152766.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152766.log.gz 2024-04-25T04:46:04.678 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61155.log 2024-04-25T04:46:04.679 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93896.log 2024-04-25T04:46:04.679 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152636.log.gz 2024-04-25T04:46:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61155.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61155.log.gz 2024-04-25T04:46:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104778.log 2024-04-25T04:46:04.680 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93896.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93896.log.gz 2024-04-25T04:46:04.681 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103301.log 2024-04-25T04:46:04.682 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104778.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104778.log.gz 2024-04-25T04:46:04.682 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116424.log 2024-04-25T04:46:04.682 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103301.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103301.log.gz 2024-04-25T04:46:04.683 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98855.log 2024-04-25T04:46:04.683 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116424.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116424.log.gz 2024-04-25T04:46:04.684 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143746.log 2024-04-25T04:46:04.684 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98855.log.gz 2024-04-25T04:46:04.684 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55616.log 2024-04-25T04:46:04.685 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143746.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143746.log.gz 2024-04-25T04:46:04.685 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123342.log 2024-04-25T04:46:04.686 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55616.log.gz 2024-04-25T04:46:04.686 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111924.log 2024-04-25T04:46:04.687 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123342.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123342.log.gz 2024-04-25T04:46:04.687 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36720.log 2024-04-25T04:46:04.688 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.111924.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111924.log.gz 2024-04-25T04:46:04.688 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124610.log 2024-04-25T04:46:04.689 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.36720.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36720.log.gz 2024-04-25T04:46:04.689 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122113.log 2024-04-25T04:46:04.690 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.124610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.124610.log.gz 2024-04-25T04:46:04.690 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50566.log 2024-04-25T04:46:04.690 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122113.log.gz 2024-04-25T04:46:04.691 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46930.log 2024-04-25T04:46:04.691 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.50566.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.50566.log.gz 2024-04-25T04:46:04.692 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83948.log 2024-04-25T04:46:04.692 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46930.log.gz 2024-04-25T04:46:04.692 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119612.log 2024-04-25T04:46:04.693 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83948.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83948.log.gz 2024-04-25T04:46:04.693 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87368.log 2024-04-25T04:46:04.694 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119612.log.gz 2024-04-25T04:46:04.697 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124677.log 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87368.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87368.log.gz 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73631.log 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.124677.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124677.log.gz 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105680.log 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73631.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73631.log.gz 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122670.log 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105680.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105680.log.gz 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106182.log 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122670.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122670.log.gz 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156511.log 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106182.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106182.log.gz 2024-04-25T04:46:04.701 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90184.log 2024-04-25T04:46:04.702 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156511.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156511.log.gz 2024-04-25T04:46:04.702 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83268.log 2024-04-25T04:46:04.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90184.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90184.log.gz 2024-04-25T04:46:04.703 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94589.log 2024-04-25T04:46:04.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83268.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83268.log.gz 2024-04-25T04:46:04.703 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118153.log 2024-04-25T04:46:04.703 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94589.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94589.log.gz 2024-04-25T04:46:04.704 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151926.log 2024-04-25T04:46:04.704 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118153.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118153.log.gz 2024-04-25T04:46:04.704 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125967.log 2024-04-25T04:46:04.705 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151926.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151926.log.gz 2024-04-25T04:46:04.705 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56414.log 2024-04-25T04:46:04.706 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125967.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125967.log.gz 2024-04-25T04:46:04.706 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142349.log 2024-04-25T04:46:04.707 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.56414.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56414.log.gz 2024-04-25T04:46:04.707 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108461.log 2024-04-25T04:46:04.707 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142349.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142349.log.gz 2024-04-25T04:46:04.708 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76882.log 2024-04-25T04:46:04.709 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108461.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108461.log.gz 2024-04-25T04:46:04.709 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91496.log 2024-04-25T04:46:04.709 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76882.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76882.log.gz 2024-04-25T04:46:04.710 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73370.log 2024-04-25T04:46:04.711 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91496.log.gz 2024-04-25T04:46:04.711 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148121.log 2024-04-25T04:46:04.711 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73370.log.gz 2024-04-25T04:46:04.711 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118282.log 2024-04-25T04:46:04.712 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148121.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148121.log.gz 2024-04-25T04:46:04.712 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48915.log 2024-04-25T04:46:04.713 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118282.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118282.log.gz 2024-04-25T04:46:04.713 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131543.log 2024-04-25T04:46:04.714 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48915.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48915.log.gz 2024-04-25T04:46:04.714 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-04-25T04:46:04.715 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131543.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131543.log.gz 2024-04-25T04:46:04.715 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58671.log 2024-04-25T04:46:04.716 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138190.log 2024-04-25T04:46:04.716 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58671.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58671.log.gz 2024-04-25T04:46:04.716 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72030.log 2024-04-25T04:46:04.717 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138190.log.gz 2024-04-25T04:46:04.718 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151621.log 2024-04-25T04:46:04.718 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72030.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72030.log.gz 2024-04-25T04:46:04.719 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78231.log 2024-04-25T04:46:04.719 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151621.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151621.log.gz 2024-04-25T04:46:04.720 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66707.log 2024-04-25T04:46:04.720 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.78231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78231.log.gz 2024-04-25T04:46:04.721 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109401.log 2024-04-25T04:46:04.721 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66707.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66707.log.gz 2024-04-25T04:46:04.721 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60981.log 2024-04-25T04:46:04.722 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109401.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109401.log.gz 2024-04-25T04:46:04.722 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117355.log 2024-04-25T04:46:04.723 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60981.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60981.log.gz 2024-04-25T04:46:04.723 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43276.log 2024-04-25T04:46:04.724 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117355.log.gz 2024-04-25T04:46:04.724 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148585.log 2024-04-25T04:46:04.725 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43276.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43276.log.gz 2024-04-25T04:46:04.725 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127180.log 2024-04-25T04:46:04.726 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148585.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148585.log.gz 2024-04-25T04:46:04.726 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119930.log 2024-04-25T04:46:04.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127180.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127180.log.gz 2024-04-25T04:46:04.727 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116402.log 2024-04-25T04:46:04.727 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119930.log.gz 2024-04-25T04:46:04.728 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91287.log 2024-04-25T04:46:04.729 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116402.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116402.log.gz 2024-04-25T04:46:04.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62787.log 2024-04-25T04:46:04.729 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91287.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91287.log.gz 2024-04-25T04:46:04.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82304.log 2024-04-25T04:46:04.730 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62787.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62787.log.gz 2024-04-25T04:46:04.730 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120282.log 2024-04-25T04:46:04.731 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82304.log.gz 2024-04-25T04:46:04.731 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53716.log 2024-04-25T04:46:04.732 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120282.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120282.log.gz 2024-04-25T04:46:04.732 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72492.log 2024-04-25T04:46:04.733 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53716.log.gz 2024-04-25T04:46:04.733 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136349.log 2024-04-25T04:46:04.733 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72492.log.gz 2024-04-25T04:46:04.734 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67151.log 2024-04-25T04:46:04.735 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136349.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136349.log.gz 2024-04-25T04:46:04.735 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55682.log 2024-04-25T04:46:04.735 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67151.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67151.log.gz 2024-04-25T04:46:04.736 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143282.log 2024-04-25T04:46:04.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55682.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55682.log.gz 2024-04-25T04:46:04.737 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106880.log 2024-04-25T04:46:04.737 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143282.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143282.log.gz 2024-04-25T04:46:04.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151995.log 2024-04-25T04:46:04.738 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106880.log.gz 2024-04-25T04:46:04.738 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58775.log 2024-04-25T04:46:04.739 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151995.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151995.log.gz 2024-04-25T04:46:04.739 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141550.log 2024-04-25T04:46:04.740 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58775.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58775.log.gz 2024-04-25T04:46:04.740 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92738.log 2024-04-25T04:46:04.741 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141550.log.gz 2024-04-25T04:46:04.741 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57321.log 2024-04-25T04:46:04.742 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92738.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92738.log.gz 2024-04-25T04:46:04.742 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82112.log 2024-04-25T04:46:04.743 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57321.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57321.log.gz 2024-04-25T04:46:04.743 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53291.log 2024-04-25T04:46:04.744 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82112.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/cluster1-client.mirror.82112.log.gz 2024-04-25T04:46:04.744 INFO:teuthology.orchestra.run.smithi188.stderr:-- /var/log/ceph/cluster1-client.mirror.58396.log 2024-04-25T04:46:04.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53291.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53291.log.gz 2024-04-25T04:46:04.746 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155062.log 2024-04-25T04:46:04.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58396.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58396.log.gz 2024-04-25T04:46:04.746 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122872.log 2024-04-25T04:46:04.746 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155062.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155062.log.gz 2024-04-25T04:46:04.747 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80487.log 2024-04-25T04:46:04.747 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122872.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122872.log.gz 2024-04-25T04:46:04.747 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112486.log 2024-04-25T04:46:04.748 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80487.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80487.log.gz 2024-04-25T04:46:04.748 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47610.log 2024-04-25T04:46:04.749 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112486.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112486.log.gz 2024-04-25T04:46:04.749 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85735.log 2024-04-25T04:46:04.750 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47610.log.gz 2024-04-25T04:46:04.750 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110398.log 2024-04-25T04:46:04.750 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85735.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85735.log.gz 2024-04-25T04:46:04.751 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57997.log 2024-04-25T04:46:04.751 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110398.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110398.log.gz 2024-04-25T04:46:04.752 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90014.log 2024-04-25T04:46:04.752 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57997.log.gz 2024-04-25T04:46:04.752 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126053.log 2024-04-25T04:46:04.753 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90014.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90014.log.gz 2024-04-25T04:46:04.753 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145340.log 2024-04-25T04:46:04.754 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126053.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126053.log.gz 2024-04-25T04:46:04.754 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136033.log 2024-04-25T04:46:04.755 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145340.log.gz 2024-04-25T04:46:04.755 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109534.log 2024-04-25T04:46:04.756 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136033.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136033.log.gz 2024-04-25T04:46:04.756 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45013.log 2024-04-25T04:46:04.757 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109534.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109534.log.gz 2024-04-25T04:46:04.757 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128060.log 2024-04-25T04:46:04.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45013.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45013.log.gz 2024-04-25T04:46:04.758 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131376.log 2024-04-25T04:46:04.758 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128060.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128060.log.gz 2024-04-25T04:46:04.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127077.log 2024-04-25T04:46:04.759 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131376.log.gz 2024-04-25T04:46:04.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141845.log 2024-04-25T04:46:04.760 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127077.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127077.log.gz 2024-04-25T04:46:04.760 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154639.log 2024-04-25T04:46:04.761 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141845.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141845.log.gz 2024-04-25T04:46:04.761 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72157.log 2024-04-25T04:46:04.762 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154639.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154639.log.gz 2024-04-25T04:46:04.762 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78477.log 2024-04-25T04:46:04.763 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72157.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72157.log.gz 2024-04-25T04:46:04.763 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45121.log 2024-04-25T04:46:04.764 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78477.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78477.log.gz 2024-04-25T04:46:04.764 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79798.log 2024-04-25T04:46:04.765 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45121.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45121.log.gz 2024-04-25T04:46:04.765 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133711.log 2024-04-25T04:46:04.766 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79798.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79798.log.gz 2024-04-25T04:46:04.766 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90148.log 2024-04-25T04:46:04.767 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133711.log.gz 2024-04-25T04:46:04.767 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102306.log 2024-04-25T04:46:04.768 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90148.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90148.log.gz 2024-04-25T04:46:04.768 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140113.log 2024-04-25T04:46:04.769 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102306.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102306.log.gz 2024-04-25T04:46:04.769 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83058.log 2024-04-25T04:46:04.769 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140113.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140113.log.gz 2024-04-25T04:46:04.770 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83848.log 2024-04-25T04:46:04.770 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83058.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83058.log.gz 2024-04-25T04:46:04.770 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139287.log 2024-04-25T04:46:04.771 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83848.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83848.log.gz 2024-04-25T04:46:04.771 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91246.log 2024-04-25T04:46:04.772 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139287.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139287.log.gz 2024-04-25T04:46:04.772 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129079.log 2024-04-25T04:46:04.773 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91246.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91246.log.gz 2024-04-25T04:46:04.773 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124428.log 2024-04-25T04:46:04.774 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129079.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129079.log.gz 2024-04-25T04:46:04.774 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128488.log 2024-04-25T04:46:04.775 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124428.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124428.log.gz 2024-04-25T04:46:04.775 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118345.log 2024-04-25T04:46:04.776 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128488.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128488.log.gz 2024-04-25T04:46:04.776 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91384.log 2024-04-25T04:46:04.777 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118345.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118345.log.gz 2024-04-25T04:46:04.777 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137794.log 2024-04-25T04:46:04.778 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69334.log 2024-04-25T04:46:04.778 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91384.log.gz 2024-04-25T04:46:04.778 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137794.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137794.log.gz 2024-04-25T04:46:04.779 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118612.log 2024-04-25T04:46:04.779 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69334.log.gz 2024-04-25T04:46:04.779 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121279.log 2024-04-25T04:46:04.780 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118612.log.gz 2024-04-25T04:46:04.780 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41770.log 2024-04-25T04:46:04.781 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121279.log.gz 2024-04-25T04:46:04.781 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122398.log 2024-04-25T04:46:04.781 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41770.log.gz 2024-04-25T04:46:04.782 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77608.log 2024-04-25T04:46:04.783 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122398.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122398.log.gz 2024-04-25T04:46:04.783 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127497.log 2024-04-25T04:46:04.784 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77608.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77608.log.gz 2024-04-25T04:46:04.784 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156647.log 2024-04-25T04:46:04.784 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127497.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127497.log.gz 2024-04-25T04:46:04.785 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41242.log 2024-04-25T04:46:04.785 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156647.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156647.log.gz 2024-04-25T04:46:04.785 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47394.log 2024-04-25T04:46:04.786 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41242.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41242.log.gz 2024-04-25T04:46:04.786 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120179.log 2024-04-25T04:46:04.787 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47394.log.gz 2024-04-25T04:46:04.787 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125483.log 2024-04-25T04:46:04.788 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120179.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120179.log.gz 2024-04-25T04:46:04.789 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.148646.log 2024-04-25T04:46:04.789 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125483.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125483.log.gz 2024-04-25T04:46:04.790 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122112.log 2024-04-25T04:46:04.790 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.148646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77494.log 2024-04-25T04:46:04.791 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.148646.log.gz 2024-04-25T04:46:04.791 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122112.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122112.log.gz 2024-04-25T04:46:04.791 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123821.log 2024-04-25T04:46:04.792 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77494.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77494.log.gz 2024-04-25T04:46:04.792 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157350.log 2024-04-25T04:46:04.793 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123821.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123821.log.gz 2024-04-25T04:46:04.793 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69631.log 2024-04-25T04:46:04.794 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157350.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157350.log.gz 2024-04-25T04:46:04.794 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81612.log 2024-04-25T04:46:04.795 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118654.log 2024-04-25T04:46:04.795 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69631.log.gz 2024-04-25T04:46:04.796 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81612.log.gz 2024-04-25T04:46:04.796 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79168.log 2024-04-25T04:46:04.797 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118654.log.gz 2024-04-25T04:46:04.797 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53525.log 2024-04-25T04:46:04.798 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79168.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79168.log.gz 2024-04-25T04:46:04.798 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130697.log 2024-04-25T04:46:04.799 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53525.log.gz 2024-04-25T04:46:04.799 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117859.log 2024-04-25T04:46:04.800 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130697.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130697.log.gz 2024-04-25T04:46:04.800 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74030.log 2024-04-25T04:46:04.801 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117859.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117859.log.gz 2024-04-25T04:46:04.801 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127539.log 2024-04-25T04:46:04.802 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.74030.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74030.log.gz 2024-04-25T04:46:04.802 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45965.log 2024-04-25T04:46:04.802 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127539.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127539.log.gz 2024-04-25T04:46:04.803 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153165.log 2024-04-25T04:46:04.803 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45965.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45965.log.gz 2024-04-25T04:46:04.803 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155440.log 2024-04-25T04:46:04.804 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153165.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153165.log.gz 2024-04-25T04:46:04.804 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85066.log 2024-04-25T04:46:04.805 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155440.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155440.log.gz 2024-04-25T04:46:04.805 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142557.log 2024-04-25T04:46:04.806 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85066.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85066.log.gz 2024-04-25T04:46:04.806 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123132.log 2024-04-25T04:46:04.807 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142557.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142557.log.gz 2024-04-25T04:46:04.807 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134111.log 2024-04-25T04:46:04.808 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123132.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123132.log.gz 2024-04-25T04:46:04.808 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153478.log 2024-04-25T04:46:04.808 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134111.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134111.log.gz 2024-04-25T04:46:04.809 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49667.log 2024-04-25T04:46:04.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153478.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153478.log.gz 2024-04-25T04:46:04.810 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73040.log 2024-04-25T04:46:04.810 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49667.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49667.log.gz 2024-04-25T04:46:04.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119229.log 2024-04-25T04:46:04.811 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73040.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73040.log.gz 2024-04-25T04:46:04.811 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92168.log 2024-04-25T04:46:04.812 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119229.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119229.log.gz 2024-04-25T04:46:04.812 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83267.log 2024-04-25T04:46:04.813 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92168.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92168.log.gz 2024-04-25T04:46:04.813 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44254.log 2024-04-25T04:46:04.814 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83267.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83267.log.gz 2024-04-25T04:46:04.814 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105721.log 2024-04-25T04:46:04.815 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44254.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44254.log.gz 2024-04-25T04:46:04.815 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160458.log 2024-04-25T04:46:04.816 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105721.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105721.log.gz 2024-04-25T04:46:04.816 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120430.log 2024-04-25T04:46:04.816 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160458.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160458.log.gz 2024-04-25T04:46:04.817 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142074.log 2024-04-25T04:46:04.818 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120430.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120430.log.gz 2024-04-25T04:46:04.818 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72830.log 2024-04-25T04:46:04.818 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142074.log.gz 2024-04-25T04:46:04.818 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103363.log 2024-04-25T04:46:04.819 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72830.log.gz 2024-04-25T04:46:04.819 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43745.log 2024-04-25T04:46:04.820 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103363.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103363.log.gz 2024-04-25T04:46:04.820 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51349.log 2024-04-25T04:46:04.821 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43745.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43745.log.gz 2024-04-25T04:46:04.821 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112433.log 2024-04-25T04:46:04.822 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51349.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51349.log.gz 2024-04-25T04:46:04.822 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58499.log 2024-04-25T04:46:04.823 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112433.log.gz 2024-04-25T04:46:04.823 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137568.log 2024-04-25T04:46:04.824 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58499.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58499.log.gz 2024-04-25T04:46:04.824 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135934.log 2024-04-25T04:46:04.825 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137568.log.gz 2024-04-25T04:46:04.825 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83125.log 2024-04-25T04:46:04.826 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135934.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135934.log.gz 2024-04-25T04:46:04.826 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83613.log 2024-04-25T04:46:04.827 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83125.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83125.log.gz 2024-04-25T04:46:04.827 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137850.log 2024-04-25T04:46:04.828 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83613.log.gz 2024-04-25T04:46:04.828 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76315.log 2024-04-25T04:46:04.829 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137850.log.gz 2024-04-25T04:46:04.829 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88783.log 2024-04-25T04:46:04.830 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76315.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76315.log.gz 2024-04-25T04:46:04.830 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133304.log 2024-04-25T04:46:04.831 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.88783.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88783.log.gz 2024-04-25T04:46:04.831 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112305.log 2024-04-25T04:46:04.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133304.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133304.log.gz 2024-04-25T04:46:04.832 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153479.log 2024-04-25T04:46:04.832 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112305.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112305.log.gz 2024-04-25T04:46:04.833 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87022.log 2024-04-25T04:46:04.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153479.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153479.log.gz 2024-04-25T04:46:04.834 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105618.log 2024-04-25T04:46:04.834 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87022.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87022.log.gz 2024-04-25T04:46:04.835 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93713.log 2024-04-25T04:46:04.835 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105618.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105618.log.gz 2024-04-25T04:46:04.836 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140375.log 2024-04-25T04:46:04.836 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93713.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93713.log.gz 2024-04-25T04:46:04.836 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.65101.log 2024-04-25T04:46:04.837 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140375.log.gz 2024-04-25T04:46:04.837 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119760.log 2024-04-25T04:46:04.838 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.65101.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.65101.log.gz 2024-04-25T04:46:04.838 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45709.log 2024-04-25T04:46:04.839 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119760.log.gz 2024-04-25T04:46:04.839 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144845.log 2024-04-25T04:46:04.840 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45709.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45709.log.gz 2024-04-25T04:46:04.840 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44253.log 2024-04-25T04:46:04.841 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69459.log 2024-04-25T04:46:04.841 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144845.log.gz 2024-04-25T04:46:04.841 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44253.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44253.log.gz 2024-04-25T04:46:04.842 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97815.log 2024-04-25T04:46:04.842 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69459.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69459.log.gz 2024-04-25T04:46:04.843 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140710.log 2024-04-25T04:46:04.844 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97815.log.gz 2024-04-25T04:46:04.844 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71194.log 2024-04-25T04:46:04.845 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140710.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140710.log.gz 2024-04-25T04:46:04.845 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72243.log 2024-04-25T04:46:04.845 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71194.log.gz 2024-04-25T04:46:04.846 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116590.log 2024-04-25T04:46:04.846 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72243.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72243.log.gz 2024-04-25T04:46:04.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58774.log 2024-04-25T04:46:04.847 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116590.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116590.log.gz 2024-04-25T04:46:04.848 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93962.log 2024-04-25T04:46:04.848 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58774.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58774.log.gz 2024-04-25T04:46:04.849 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54793.log 2024-04-25T04:46:04.849 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93962.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93962.log.gz 2024-04-25T04:46:04.849 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40242.log 2024-04-25T04:46:04.850 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.54793.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54793.log.gz 2024-04-25T04:46:04.850 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153936.log 2024-04-25T04:46:04.851 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40242.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40242.log.gz 2024-04-25T04:46:04.851 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152074.log 2024-04-25T04:46:04.852 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153936.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153936.log.gz 2024-04-25T04:46:04.852 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98673.log 2024-04-25T04:46:04.853 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152074.log.gz 2024-04-25T04:46:04.853 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124449.log 2024-04-25T04:46:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98673.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98673.log.gz 2024-04-25T04:46:04.854 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127540.log 2024-04-25T04:46:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.124449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124449.log.gz 2024-04-25T04:46:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157032.log 2024-04-25T04:46:04.855 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127540.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127540.log.gz 2024-04-25T04:46:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42374.log 2024-04-25T04:46:04.856 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157032.log.gz 2024-04-25T04:46:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127666.log 2024-04-25T04:46:04.857 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42374.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42374.log.gz 2024-04-25T04:46:04.858 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119548.log 2024-04-25T04:46:04.858 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127666.log.gz 2024-04-25T04:46:04.859 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120284.log 2024-04-25T04:46:04.859 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119548.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119548.log.gz 2024-04-25T04:46:04.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76629.log 2024-04-25T04:46:04.860 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120284.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120284.log.gz 2024-04-25T04:46:04.860 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67367.log 2024-04-25T04:46:04.861 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76629.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76629.log.gz 2024-04-25T04:46:04.862 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155564.log 2024-04-25T04:46:04.862 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67367.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67367.log.gz 2024-04-25T04:46:04.863 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68135.log 2024-04-25T04:46:04.863 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155564.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155564.log.gz 2024-04-25T04:46:04.863 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66241.log 2024-04-25T04:46:04.864 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68135.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68135.log.gz 2024-04-25T04:46:04.864 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97758.log 2024-04-25T04:46:04.865 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66241.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66241.log.gz 2024-04-25T04:46:04.865 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76316.log 2024-04-25T04:46:04.866 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97758.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97758.log.gz 2024-04-25T04:46:04.866 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109395.log 2024-04-25T04:46:04.867 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76316.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76316.log.gz 2024-04-25T04:46:04.867 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137014.log 2024-04-25T04:46:04.868 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109395.log.gz 2024-04-25T04:46:04.868 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117103.log 2024-04-25T04:46:04.868 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137014.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137014.log.gz 2024-04-25T04:46:04.869 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107790.log 2024-04-25T04:46:04.870 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117103.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117103.log.gz 2024-04-25T04:46:04.870 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133248.log 2024-04-25T04:46:04.870 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107790.log.gz 2024-04-25T04:46:04.871 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152155.log 2024-04-25T04:46:04.871 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133248.log.gz 2024-04-25T04:46:04.872 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133079.log 2024-04-25T04:46:04.872 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152155.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152155.log.gz 2024-04-25T04:46:04.872 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106042.log 2024-04-25T04:46:04.873 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133079.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133079.log.gz 2024-04-25T04:46:04.873 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75387.log 2024-04-25T04:46:04.874 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106042.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106042.log.gz 2024-04-25T04:46:04.874 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71144.log 2024-04-25T04:46:04.875 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75387.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75387.log.gz 2024-04-25T04:46:04.875 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100568.log 2024-04-25T04:46:04.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71144.log.gz 2024-04-25T04:46:04.876 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128849.log 2024-04-25T04:46:04.876 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100568.log.gz 2024-04-25T04:46:04.877 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121242.log 2024-04-25T04:46:04.877 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128849.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128849.log.gz 2024-04-25T04:46:04.877 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47695.log 2024-04-25T04:46:04.878 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121242.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121242.log.gz 2024-04-25T04:46:04.878 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148255.log 2024-04-25T04:46:04.879 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47695.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47695.log.gz 2024-04-25T04:46:04.879 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151768.log 2024-04-25T04:46:04.880 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148255.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148255.log.gz 2024-04-25T04:46:04.880 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133600.log 2024-04-25T04:46:04.881 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151768.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151768.log.gz 2024-04-25T04:46:04.881 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130910.log 2024-04-25T04:46:04.882 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133600.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133600.log.gz 2024-04-25T04:46:04.882 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97810.log 2024-04-25T04:46:04.883 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130910.log.gz 2024-04-25T04:46:04.883 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116045.log 2024-04-25T04:46:04.884 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97810.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97810.log.gz 2024-04-25T04:46:04.884 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131079.log 2024-04-25T04:46:04.884 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116045.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116045.log.gz 2024-04-25T04:46:04.885 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133166.log 2024-04-25T04:46:04.885 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131079.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131079.log.gz 2024-04-25T04:46:04.885 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112764.log 2024-04-25T04:46:04.886 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133166.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133166.log.gz 2024-04-25T04:46:04.886 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53411.log 2024-04-25T04:46:04.887 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.112764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.112764.log.gz 2024-04-25T04:46:04.887 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65811.log 2024-04-25T04:46:04.888 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53411.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53411.log.gz 2024-04-25T04:46:04.888 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143199.log 2024-04-25T04:46:04.889 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.65811.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65811.log.gz 2024-04-25T04:46:04.889 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45669.log 2024-04-25T04:46:04.890 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143199.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143199.log.gz 2024-04-25T04:46:04.890 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76681.log 2024-04-25T04:46:04.891 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45669.log.gz 2024-04-25T04:46:04.891 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92867.log 2024-04-25T04:46:04.892 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.76681.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76681.log.gz 2024-04-25T04:46:04.892 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144374.log 2024-04-25T04:46:04.893 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92867.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92867.log.gz 2024-04-25T04:46:04.893 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92167.log 2024-04-25T04:46:04.894 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144374.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144374.log.gz 2024-04-25T04:46:04.894 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153233.log 2024-04-25T04:46:04.894 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92167.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92167.log.gz 2024-04-25T04:46:04.895 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46025.log 2024-04-25T04:46:04.895 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153233.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153233.log.gz 2024-04-25T04:46:04.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85376.log 2024-04-25T04:46:04.896 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46025.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46025.log.gz 2024-04-25T04:46:04.896 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111272.log 2024-04-25T04:46:04.897 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85376.log.gz 2024-04-25T04:46:04.897 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123595.log 2024-04-25T04:46:04.898 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111272.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111272.log.gz 2024-04-25T04:46:04.898 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118426.log 2024-04-25T04:46:04.903 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123595.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123595.log.gz 2024-04-25T04:46:04.903 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148235.log 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118426.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118426.log.gz 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63701.log 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148235.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148235.log.gz 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69172.log 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63701.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63701.log.gz 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118194.log 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69172.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69172.log.gz 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141507.log 2024-04-25T04:46:04.904 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118194.log.gz 2024-04-25T04:46:04.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125811.log 2024-04-25T04:46:04.905 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141507.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141507.log.gz 2024-04-25T04:46:04.905 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74525.log 2024-04-25T04:46:04.906 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125811.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125811.log.gz 2024-04-25T04:46:04.906 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99156.log 2024-04-25T04:46:04.906 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74525.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74525.log.gz 2024-04-25T04:46:04.906 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52568.log 2024-04-25T04:46:04.907 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.99156.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99156.log.gz 2024-04-25T04:46:04.907 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159552.log 2024-04-25T04:46:04.908 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52568.log.gz 2024-04-25T04:46:04.908 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68047.log 2024-04-25T04:46:04.909 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159552.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159552.log.gz 2024-04-25T04:46:04.909 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83716.log 2024-04-25T04:46:04.910 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68047.log.gz 2024-04-25T04:46:04.910 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126721.log 2024-04-25T04:46:04.910 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83716.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83716.log.gz 2024-04-25T04:46:04.911 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58607.log 2024-04-25T04:46:04.911 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126721.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126721.log.gz 2024-04-25T04:46:04.911 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160519.log 2024-04-25T04:46:04.912 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58607.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58607.log.gz 2024-04-25T04:46:04.912 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130717.log 2024-04-25T04:46:04.913 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160519.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160519.log.gz 2024-04-25T04:46:04.913 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109614.log 2024-04-25T04:46:04.914 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130717.log.gz 2024-04-25T04:46:04.914 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126346.log 2024-04-25T04:46:04.915 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.109614.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109614.log.gz 2024-04-25T04:46:04.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46635.log 2024-04-25T04:46:04.916 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126346.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126346.log.gz 2024-04-25T04:46:04.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125881.log 2024-04-25T04:46:04.917 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46635.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46635.log.gz 2024-04-25T04:46:04.917 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43605.log 2024-04-25T04:46:04.918 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125881.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125881.log.gz 2024-04-25T04:46:04.918 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62189.log 2024-04-25T04:46:04.919 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43605.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43605.log.gz 2024-04-25T04:46:04.919 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143536.log 2024-04-25T04:46:04.920 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62189.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62189.log.gz 2024-04-25T04:46:04.920 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66327.log 2024-04-25T04:46:04.920 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143536.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143536.log.gz 2024-04-25T04:46:04.921 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65899.log 2024-04-25T04:46:04.921 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66327.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66327.log.gz 2024-04-25T04:46:04.921 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64836.log 2024-04-25T04:46:04.922 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65899.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65899.log.gz 2024-04-25T04:46:04.922 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108486.log 2024-04-25T04:46:04.923 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.64836.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64836.log.gz 2024-04-25T04:46:04.923 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116549.log 2024-04-25T04:46:04.924 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108486.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108486.log.gz 2024-04-25T04:46:04.924 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102458.log 2024-04-25T04:46:04.925 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116549.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116549.log.gz 2024-04-25T04:46:04.925 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62232.log 2024-04-25T04:46:04.926 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102458.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102458.log.gz 2024-04-25T04:46:04.926 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127729.log 2024-04-25T04:46:04.926 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62232.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62232.log.gz 2024-04-25T04:46:04.927 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123423.log 2024-04-25T04:46:04.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127729.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127729.log.gz 2024-04-25T04:46:04.928 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72115.log 2024-04-25T04:46:04.928 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123423.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123423.log.gz 2024-04-25T04:46:04.929 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43998.log 2024-04-25T04:46:04.929 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72115.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72115.log.gz 2024-04-25T04:46:04.929 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.136900.log 2024-04-25T04:46:04.930 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43998.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43998.log.gz 2024-04-25T04:46:04.930 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105974.log 2024-04-25T04:46:04.931 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.136900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.136900.log.gz 2024-04-25T04:46:04.931 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69606.log 2024-04-25T04:46:04.932 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105974.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105974.log.gz 2024-04-25T04:46:04.932 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83741.log 2024-04-25T04:46:04.933 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69606.log.gz 2024-04-25T04:46:04.933 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154150.log 2024-04-25T04:46:04.934 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83741.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83741.log.gz 2024-04-25T04:46:04.934 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143032.log 2024-04-25T04:46:04.935 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154150.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154150.log.gz 2024-04-25T04:46:04.935 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143704.log 2024-04-25T04:46:04.936 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143032.log.gz 2024-04-25T04:46:04.936 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136302.log 2024-04-25T04:46:04.937 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143704.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143704.log.gz 2024-04-25T04:46:04.937 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61407.log 2024-04-25T04:46:04.937 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136302.log.gz 2024-04-25T04:46:04.938 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117522.log 2024-04-25T04:46:04.938 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61407.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61407.log.gz 2024-04-25T04:46:04.939 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119149.log 2024-04-25T04:46:04.939 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117522.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117522.log.gz 2024-04-25T04:46:04.940 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134918.log 2024-04-25T04:46:04.940 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119149.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119149.log.gz 2024-04-25T04:46:04.940 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127119.log 2024-04-25T04:46:04.941 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134918.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134918.log.gz 2024-04-25T04:46:04.941 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143617.log 2024-04-25T04:46:04.942 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127119.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127119.log.gz 2024-04-25T04:46:04.942 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55306.log 2024-04-25T04:46:04.943 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143617.log.gz 2024-04-25T04:46:04.943 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108047.log 2024-04-25T04:46:04.944 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55306.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55306.log.gz 2024-04-25T04:46:04.944 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94383.log 2024-04-25T04:46:04.945 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108047.log.gz 2024-04-25T04:46:04.945 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52847.log 2024-04-25T04:46:04.946 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94383.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94383.log.gz 2024-04-25T04:46:04.946 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151117.log 2024-04-25T04:46:04.947 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52847.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52847.log.gz 2024-04-25T04:46:04.947 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130023.log 2024-04-25T04:46:04.948 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52395.log 2024-04-25T04:46:04.948 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151117.log.gz 2024-04-25T04:46:04.948 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130023.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130023.log.gz 2024-04-25T04:46:04.948 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73251.log 2024-04-25T04:46:04.949 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52395.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52395.log.gz 2024-04-25T04:46:04.950 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81129.log 2024-04-25T04:46:04.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41049.log 2024-04-25T04:46:04.951 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73251.log.gz 2024-04-25T04:46:04.951 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81129.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81129.log.gz 2024-04-25T04:46:04.952 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83124.log 2024-04-25T04:46:04.952 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41049.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41049.log.gz 2024-04-25T04:46:04.952 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115712.log 2024-04-25T04:46:04.953 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83124.log.gz 2024-04-25T04:46:04.953 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143136.log 2024-04-25T04:46:04.954 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115712.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115712.log.gz 2024-04-25T04:46:04.954 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58542.log 2024-04-25T04:46:04.955 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143136.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143136.log.gz 2024-04-25T04:46:04.955 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140374.log 2024-04-25T04:46:04.956 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58542.log.gz 2024-04-25T04:46:04.956 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108169.log 2024-04-25T04:46:04.957 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140374.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140374.log.gz 2024-04-25T04:46:04.957 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121393.log 2024-04-25T04:46:04.958 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108169.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108169.log.gz 2024-04-25T04:46:04.958 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53269.log 2024-04-25T04:46:04.959 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121393.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121393.log.gz 2024-04-25T04:46:04.959 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134769.log 2024-04-25T04:46:04.960 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53269.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53269.log.gz 2024-04-25T04:46:04.960 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145896.log 2024-04-25T04:46:04.961 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.134769.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134769.log.gz 2024-04-25T04:46:04.961 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111749.log 2024-04-25T04:46:04.962 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145896.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145896.log.gz 2024-04-25T04:46:04.962 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53323.log 2024-04-25T04:46:04.963 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111749.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111749.log.gz 2024-04-25T04:46:04.963 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124636.log 2024-04-25T04:46:04.963 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53323.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.53323.log.gz 2024-04-25T04:46:04.964 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54332.log 2024-04-25T04:46:04.964 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124636.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124636.log.gz 2024-04-25T04:46:04.964 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71392.log 2024-04-25T04:46:04.965 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.54332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54332.log.gz 2024-04-25T04:46:04.965 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112478.log 2024-04-25T04:46:04.966 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71392.log.gz 2024-04-25T04:46:04.966 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84163.log 2024-04-25T04:46:04.967 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112478.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112478.log.gz 2024-04-25T04:46:04.967 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105053.log 2024-04-25T04:46:04.968 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84163.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84163.log.gz 2024-04-25T04:46:04.968 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110682.log 2024-04-25T04:46:04.969 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105053.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105053.log.gz 2024-04-25T04:46:04.969 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157191.log 2024-04-25T04:46:04.970 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110682.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110682.log.gz 2024-04-25T04:46:04.970 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74005.log 2024-04-25T04:46:04.971 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157191.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157191.log.gz 2024-04-25T04:46:04.971 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121236.log 2024-04-25T04:46:04.972 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.74005.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74005.log.gz 2024-04-25T04:46:04.972 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144857.log 2024-04-25T04:46:04.973 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121236.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121236.log.gz 2024-04-25T04:46:04.973 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44207.log 2024-04-25T04:46:04.974 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.144857.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144857.log.gz 2024-04-25T04:46:04.974 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135957.log 2024-04-25T04:46:04.974 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44207.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44207.log.gz 2024-04-25T04:46:04.975 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102903.log 2024-04-25T04:46:04.975 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135957.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135957.log.gz 2024-04-25T04:46:04.976 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82557.log 2024-04-25T04:46:04.976 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102903.log.gz 2024-04-25T04:46:04.976 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79075.log 2024-04-25T04:46:04.977 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82557.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82557.log.gz 2024-04-25T04:46:04.977 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113140.log 2024-04-25T04:46:04.978 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79075.log.gz 2024-04-25T04:46:04.978 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155101.log 2024-04-25T04:46:04.979 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.113140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113140.log.gz 2024-04-25T04:46:04.979 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102552.log 2024-04-25T04:46:04.979 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155101.log.gz 2024-04-25T04:46:04.980 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106312.log 2024-04-25T04:46:04.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102552.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102552.log.gz 2024-04-25T04:46:04.981 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114717.log 2024-04-25T04:46:04.981 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106312.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106312.log.gz 2024-04-25T04:46:04.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139852.log 2024-04-25T04:46:04.982 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114717.log.gz 2024-04-25T04:46:04.982 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142391.log 2024-04-25T04:46:04.983 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139852.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139852.log.gz 2024-04-25T04:46:04.984 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68406.log 2024-04-25T04:46:04.984 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142391.log.gz 2024-04-25T04:46:04.985 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116252.log 2024-04-25T04:46:04.985 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68406.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68406.log.gz 2024-04-25T04:46:04.985 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120200.log 2024-04-25T04:46:04.986 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116252.log.gz 2024-04-25T04:46:04.987 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42964.log 2024-04-25T04:46:04.987 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120200.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120200.log.gz 2024-04-25T04:46:04.987 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143200.log 2024-04-25T04:46:04.988 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42964.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42964.log.gz 2024-04-25T04:46:04.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41804.log 2024-04-25T04:46:04.989 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143200.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.143200.log.gz-5 2024-04-25T04:46:04.989 INFO:teuthology.orchestra.run.smithi188.stderr:--verbose -- /var/log/ceph/cluster1-client.admin.157571.log 2024-04-25T04:46:04.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.41804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41804.log.gz 2024-04-25T04:46:04.990 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70535.log 2024-04-25T04:46:04.990 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157571.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157571.log.gz 2024-04-25T04:46:04.991 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47759.log 2024-04-25T04:46:04.992 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70535.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70535.log.gz 2024-04-25T04:46:04.992 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93563.log 2024-04-25T04:46:04.992 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47759.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47759.log.gz 2024-04-25T04:46:04.992 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69948.log 2024-04-25T04:46:04.993 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93563.log.gz 2024-04-25T04:46:04.993 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45919.log 2024-04-25T04:46:04.994 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69948.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69948.log.gz 2024-04-25T04:46:04.994 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94316.log 2024-04-25T04:46:04.995 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45919.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45919.log.gz 2024-04-25T04:46:04.995 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156637.log 2024-04-25T04:46:04.996 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94316.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94316.log.gz 2024-04-25T04:46:04.996 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153814.log 2024-04-25T04:46:04.997 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156637.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156637.log.gz 2024-04-25T04:46:04.997 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60608.log 2024-04-25T04:46:04.998 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153814.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153814.log.gz 2024-04-25T04:46:04.998 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118154.log 2024-04-25T04:46:04.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60608.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60608.log.gz 2024-04-25T04:46:04.999 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117610.log 2024-04-25T04:46:04.999 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118154.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118154.log.gz 2024-04-25T04:46:05.000 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85355.log 2024-04-25T04:46:05.000 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117610.log.gz 2024-04-25T04:46:05.000 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119464.log 2024-04-25T04:46:05.001 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85355.log.gz 2024-04-25T04:46:05.001 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67212.log 2024-04-25T04:46:05.002 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119464.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119464.log.gz 2024-04-25T04:46:05.002 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145765.log 2024-04-25T04:46:05.003 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67212.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67212.log.gz 2024-04-25T04:46:05.003 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75952.log 2024-04-25T04:46:05.004 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145765.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145765.log.gz 2024-04-25T04:46:05.004 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158213.log 2024-04-25T04:46:05.005 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75952.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75952.log.gz 2024-04-25T04:46:05.005 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121277.log 2024-04-25T04:46:05.006 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158213.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158213.log.gz 2024-04-25T04:46:05.006 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157918.log 2024-04-25T04:46:05.007 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52987.log 2024-04-25T04:46:05.007 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121277.log.gz 2024-04-25T04:46:05.008 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157918.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157918.log.gz 2024-04-25T04:46:05.008 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67762.log 2024-04-25T04:46:05.008 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52987.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52987.log.gz 2024-04-25T04:46:05.009 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66636.log 2024-04-25T04:46:05.009 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.67762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67762.log.gz 2024-04-25T04:46:05.009 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.100925.log 2024-04-25T04:46:05.010 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66636.log.gz 2024-04-25T04:46:05.010 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151076.log 2024-04-25T04:46:05.011 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.100925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.100925.log.gz 2024-04-25T04:46:05.011 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127604.log 2024-04-25T04:46:05.012 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151076.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151076.log.gz 2024-04-25T04:46:05.012 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118492.log 2024-04-25T04:46:05.013 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127604.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127604.log.gz 2024-04-25T04:46:05.013 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145720.log 2024-04-25T04:46:05.014 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118492.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118492.log.gz 2024-04-25T04:46:05.014 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83742.log 2024-04-25T04:46:05.015 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145720.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145720.log.gz 2024-04-25T04:46:05.015 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41702.log 2024-04-25T04:46:05.016 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83742.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83742.log.gz 2024-04-25T04:46:05.016 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44106.log 2024-04-25T04:46:05.017 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.41702.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41702.log.gz 2024-04-25T04:46:05.017 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95520.log 2024-04-25T04:46:05.018 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44106.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44106.log.gz 2024-04-25T04:46:05.018 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72954.log 2024-04-25T04:46:05.019 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95520.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95520.log.gz 2024-04-25T04:46:05.019 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107623.log 2024-04-25T04:46:05.019 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72954.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72954.log.gz 2024-04-25T04:46:05.019 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145612.log 2024-04-25T04:46:05.020 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107623.log.gz 2024-04-25T04:46:05.020 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107553.log 2024-04-25T04:46:05.021 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145612.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145612.log.gz 2024-04-25T04:46:05.021 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139414.log 2024-04-25T04:46:05.022 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107553.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107553.log.gz 2024-04-25T04:46:05.022 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78279.log 2024-04-25T04:46:05.023 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139414.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139414.log.gz 2024-04-25T04:46:05.023 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112790.log 2024-04-25T04:46:05.024 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78279.log.gz 2024-04-25T04:46:05.024 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151389.log 2024-04-25T04:46:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112790.log.gz 2024-04-25T04:46:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59428.log 2024-04-25T04:46:05.025 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151389.log.gz 2024-04-25T04:46:05.026 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110462.log 2024-04-25T04:46:05.026 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59428.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59428.log.gz 2024-04-25T04:46:05.027 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135493.log 2024-04-25T04:46:05.027 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110462.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110462.log.gz 2024-04-25T04:46:05.027 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106775.log 2024-04-25T04:46:05.028 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135493.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135493.log.gz 2024-04-25T04:46:05.028 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40866.log 2024-04-25T04:46:05.029 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106775.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106775.log.gz 2024-04-25T04:46:05.029 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.64421.log 2024-04-25T04:46:05.030 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40866.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40866.log.gz 2024-04-25T04:46:05.030 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136074.log 2024-04-25T04:46:05.031 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.64421.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.64421.log.gz 2024-04-25T04:46:05.031 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128125.log 2024-04-25T04:46:05.032 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136074.log.gz 2024-04-25T04:46:05.032 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80865.log 2024-04-25T04:46:05.033 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128125.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128125.log.gz 2024-04-25T04:46:05.033 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139896.log 2024-04-25T04:46:05.034 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80865.log.gz 2024-04-25T04:46:05.034 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70805.log 2024-04-25T04:46:05.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139896.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139896.log.gz 2024-04-25T04:46:05.035 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160108.log 2024-04-25T04:46:05.035 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70805.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70805.log.gz 2024-04-25T04:46:05.036 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94737.log 2024-04-25T04:46:05.036 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160108.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160108.log.gz 2024-04-25T04:46:05.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70449.log 2024-04-25T04:46:05.038 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80930.log 2024-04-25T04:46:05.038 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94737.log.gz 2024-04-25T04:46:05.038 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70449.log.gz 2024-04-25T04:46:05.038 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106020.log 2024-04-25T04:46:05.039 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80930.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80930.log.gz 2024-04-25T04:46:05.039 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87049.log 2024-04-25T04:46:05.040 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106020.log.gz 2024-04-25T04:46:05.040 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118531.log 2024-04-25T04:46:05.041 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87049.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87049.log.gz 2024-04-25T04:46:05.041 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46528.log 2024-04-25T04:46:05.042 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118531.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118531.log.gz 2024-04-25T04:46:05.042 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112582.log 2024-04-25T04:46:05.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46528.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46528.log.gz 2024-04-25T04:46:05.043 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76574.log 2024-04-25T04:46:05.043 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.112582.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112582.log.gz 2024-04-25T04:46:05.044 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125275.log 2024-04-25T04:46:05.044 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76574.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76574.log.gz 2024-04-25T04:46:05.044 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72535.log 2024-04-25T04:46:05.045 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.125275.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.125275.log.gz 2024-04-25T04:46:05.045 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72200.log 2024-04-25T04:46:05.046 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72535.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72535.log.gz 2024-04-25T04:46:05.046 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104717.log 2024-04-25T04:46:05.047 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.72200.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72200.log.gz 2024-04-25T04:46:05.047 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102792.log 2024-04-25T04:46:05.048 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104717.log.gz 2024-04-25T04:46:05.048 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101975.log 2024-04-25T04:46:05.049 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102792.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102792.log.gz 2024-04-25T04:46:05.049 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110330.log 2024-04-25T04:46:05.050 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101975.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101975.log.gz 2024-04-25T04:46:05.050 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93852.log 2024-04-25T04:46:05.051 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110330.log.gz 2024-04-25T04:46:05.051 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118535.log 2024-04-25T04:46:05.052 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93852.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93852.log.gz 2024-04-25T04:46:05.052 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145012.log 2024-04-25T04:46:05.052 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118535.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118535.log.gz 2024-04-25T04:46:05.053 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126086.log 2024-04-25T04:46:05.053 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145012.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145012.log.gz 2024-04-25T04:46:05.054 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80321.log 2024-04-25T04:46:05.054 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126086.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126086.log.gz 2024-04-25T04:46:05.054 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81611.log 2024-04-25T04:46:05.055 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80321.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80321.log.gz 2024-04-25T04:46:05.055 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90318.log 2024-04-25T04:46:05.056 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81611.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81611.log.gz 2024-04-25T04:46:05.056 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123343.log 2024-04-25T04:46:05.057 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90318.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90318.log.gz 2024-04-25T04:46:05.057 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142476.log 2024-04-25T04:46:05.058 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123343.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123343.log.gz 2024-04-25T04:46:05.058 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45588.log 2024-04-25T04:46:05.059 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142476.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142476.log.gz 2024-04-25T04:46:05.059 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97809.log 2024-04-25T04:46:05.060 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45588.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45588.log.gz 2024-04-25T04:46:05.060 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121320.log 2024-04-25T04:46:05.061 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.97809.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97809.log.gz 2024-04-25T04:46:05.061 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49163.log 2024-04-25T04:46:05.061 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121320.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121320.log.gz 2024-04-25T04:46:05.061 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121264.log 2024-04-25T04:46:05.062 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49163.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49163.log.gz 2024-04-25T04:46:05.063 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94904.log 2024-04-25T04:46:05.063 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121264.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121264.log.gz 2024-04-25T04:46:05.063 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128416.log 2024-04-25T04:46:05.064 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94904.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94904.log.gz 2024-04-25T04:46:05.064 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133626.log 2024-04-25T04:46:05.065 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128416.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128416.log.gz 2024-04-25T04:46:05.065 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109451.log 2024-04-25T04:46:05.066 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133626.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133626.log.gz 2024-04-25T04:46:05.066 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43580.log 2024-04-25T04:46:05.067 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109451.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109451.log.gz 2024-04-25T04:46:05.067 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107912.log 2024-04-25T04:46:05.068 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43580.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43580.log.gz 2024-04-25T04:46:05.068 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140872.log 2024-04-25T04:46:05.069 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107912.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107912.log.gz 2024-04-25T04:46:05.070 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51107.log 2024-04-25T04:46:05.070 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140872.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140872.log.gz 2024-04-25T04:46:05.070 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41994.log 2024-04-25T04:46:05.070 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.51107.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51107.log.gz 2024-04-25T04:46:05.071 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127727.log 2024-04-25T04:46:05.071 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.41994.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41994.log.gz 2024-04-25T04:46:05.072 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124750.log 2024-04-25T04:46:05.073 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.127727.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127727.log.gz 2024-04-25T04:46:05.073 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82239.log 2024-04-25T04:46:05.073 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124750.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124750.log.gz 2024-04-25T04:46:05.074 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150084.log 2024-04-25T04:46:05.074 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82239.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82239.log.gz 2024-04-25T04:46:05.075 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89293.log 2024-04-25T04:46:05.075 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150084.log.gz 2024-04-25T04:46:05.076 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82617.log 2024-04-25T04:46:05.076 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89293.log: gzip 0.0%-5 -- replaced with /var/log/ceph/cluster2-client.mirror.89293.log.gz 2024-04-25T04:46:05.077 INFO:teuthology.orchestra.run.smithi188.stderr:--verbose -- /var/log/ceph/cluster1-client.mirror.132340.log 2024-04-25T04:46:05.077 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82617.log.gz 2024-04-25T04:46:05.077 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46719.log 2024-04-25T04:46:05.078 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132340.log.gz 2024-04-25T04:46:05.078 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131001.log 2024-04-25T04:46:05.079 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46719.log.gz 2024-04-25T04:46:05.079 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55344.log 2024-04-25T04:46:05.080 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131001.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131001.log.gz 2024-04-25T04:46:05.080 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49907.log 2024-04-25T04:46:05.081 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55344.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55344.log.gz 2024-04-25T04:46:05.081 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53452.log 2024-04-25T04:46:05.082 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.49907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.49907.log.gz 2024-04-25T04:46:05.082 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108542.log 2024-04-25T04:46:05.083 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53452.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53452.log.gz 2024-04-25T04:46:05.083 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57084.log 2024-04-25T04:46:05.083 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108542.log.gz 2024-04-25T04:46:05.084 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129762.log 2024-04-25T04:46:05.084 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57084.log.gz 2024-04-25T04:46:05.085 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105532.log 2024-04-25T04:46:05.085 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129762.log.gz 2024-04-25T04:46:05.086 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.123658.log 2024-04-25T04:46:05.086 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66706.log 2024-04-25T04:46:05.087 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105532.log.gz 2024-04-25T04:46:05.087 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.123658.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.123658.log.gz 2024-04-25T04:46:05.087 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90477.log 2024-04-25T04:46:05.088 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.66706.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66706.log.gz 2024-04-25T04:46:05.088 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157549.log 2024-04-25T04:46:05.089 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90477.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90477.log.gz 2024-04-25T04:46:05.089 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159357.log 2024-04-25T04:46:05.090 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157549.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157549.log.gz 2024-04-25T04:46:05.090 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117061.log 2024-04-25T04:46:05.091 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159357.log.gz 2024-04-25T04:46:05.091 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.101442.log 2024-04-25T04:46:05.091 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117061.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117061.log.gz 2024-04-25T04:46:05.092 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150764.log 2024-04-25T04:46:05.092 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.101442.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.101442.log.gz 2024-04-25T04:46:05.093 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112454.log 2024-04-25T04:46:05.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150764.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150764.log.gz 2024-04-25T04:46:05.094 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81130.log 2024-04-25T04:46:05.094 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112454.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112454.log.gz 2024-04-25T04:46:05.095 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139643.log 2024-04-25T04:46:05.095 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81130.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81130.log.gz 2024-04-25T04:46:05.095 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99510.log 2024-04-25T04:46:05.096 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139643.log.gz 2024-04-25T04:46:05.096 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77520.log 2024-04-25T04:46:05.097 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99510.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99510.log.gz 2024-04-25T04:46:05.097 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81736.log 2024-04-25T04:46:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.77520.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77520.log.gz 2024-04-25T04:46:05.098 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85067.log 2024-04-25T04:46:05.099 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81736.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81736.log.gz 2024-04-25T04:46:05.099 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154597.log 2024-04-25T04:46:05.100 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.85067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85067.log.gz 2024-04-25T04:46:05.100 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57144.log 2024-04-25T04:46:05.101 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154597.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154597.log.gz 2024-04-25T04:46:05.101 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126384.log 2024-04-25T04:46:05.102 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57144.log.gz 2024-04-25T04:46:05.102 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69505.log 2024-04-25T04:46:05.106 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.126384.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126384.log.gz 2024-04-25T04:46:05.106 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90606.log 2024-04-25T04:46:05.106 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69505.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69505.log.gz 2024-04-25T04:46:05.106 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47987.log 2024-04-25T04:46:05.106 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90606.log.gz 2024-04-25T04:46:05.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110616.log 2024-04-25T04:46:05.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47987.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47987.log.gz 2024-04-25T04:46:05.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154004.log 2024-04-25T04:46:05.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110616.log.gz 2024-04-25T04:46:05.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87358.log 2024-04-25T04:46:05.107 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154004.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154004.log.gz 2024-04-25T04:46:05.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136509.log 2024-04-25T04:46:05.108 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87358.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87358.log.gz 2024-04-25T04:46:05.109 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78922.log 2024-04-25T04:46:05.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.136509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150823.log 2024-04-25T04:46:05.110 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136509.log.gz 2024-04-25T04:46:05.110 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78922.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78922.log.gz 2024-04-25T04:46:05.111 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48573.log 2024-04-25T04:46:05.111 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150823.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150823.log.gz 2024-04-25T04:46:05.112 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61555.log 2024-04-25T04:46:05.112 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48573.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48573.log.gz 2024-04-25T04:46:05.112 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94802.log 2024-04-25T04:46:05.113 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61555.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61555.log.gz 2024-04-25T04:46:05.113 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141211.log 2024-04-25T04:46:05.114 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94802.log.gz 2024-04-25T04:46:05.114 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144186.log 2024-04-25T04:46:05.115 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141211.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141211.log.gz 2024-04-25T04:46:05.115 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88140.log 2024-04-25T04:46:05.116 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144186.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144186.log.gz 2024-04-25T04:46:05.116 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.160686.log 2024-04-25T04:46:05.116 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88140.log.gz 2024-04-25T04:46:05.117 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83906.log 2024-04-25T04:46:05.118 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.160686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.41037.log 2024-04-25T04:46:05.118 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.160686.log.gz 2024-04-25T04:46:05.119 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83906.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83906.log.gz 2024-04-25T04:46:05.119 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145523.log 2024-04-25T04:46:05.120 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.41037.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.41037.log.gz 2024-04-25T04:46:05.120 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47265.log 2024-04-25T04:46:05.120 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145523.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145523.log.gz 2024-04-25T04:46:05.121 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100560.log 2024-04-25T04:46:05.122 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156526.log 2024-04-25T04:46:05.122 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47265.log.gz 2024-04-25T04:46:05.123 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100560.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100560.log.gz 2024-04-25T04:46:05.123 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69923.log 2024-04-25T04:46:05.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156526.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156526.log.gz 2024-04-25T04:46:05.124 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78317.log 2024-04-25T04:46:05.124 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69923.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69923.log.gz 2024-04-25T04:46:05.124 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111763.log 2024-04-25T04:46:05.125 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78317.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78317.log.gz 2024-04-25T04:46:05.125 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114428.log 2024-04-25T04:46:05.126 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.111763.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111763.log.gz 2024-04-25T04:46:05.126 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45500.log 2024-04-25T04:46:05.127 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.114428.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114428.log.gz 2024-04-25T04:46:05.127 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135026.log 2024-04-25T04:46:05.128 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45500.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45500.log.gz 2024-04-25T04:46:05.128 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108695.log 2024-04-25T04:46:05.129 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135026.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135026.log.gz 2024-04-25T04:46:05.129 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146017.log 2024-04-25T04:46:05.130 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108695.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108695.log.gz 2024-04-25T04:46:05.130 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53619.log 2024-04-25T04:46:05.131 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146017.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146017.log.gz 2024-04-25T04:46:05.131 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94502.log 2024-04-25T04:46:05.131 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53619.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53619.log.gz 2024-04-25T04:46:05.132 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46995.log 2024-04-25T04:46:05.133 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94502.log.gz 2024-04-25T04:46:05.133 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68760.log 2024-04-25T04:46:05.134 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46995.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46995.log.gz 2024-04-25T04:46:05.134 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.135743.log 2024-04-25T04:46:05.134 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68760.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68760.log.gz 2024-04-25T04:46:05.134 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140457.log 2024-04-25T04:46:05.135 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.135743.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.135743.log.gz 2024-04-25T04:46:05.135 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.75922.log 2024-04-25T04:46:05.136 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.140457.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140457.log.gz 2024-04-25T04:46:05.136 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44732.log 2024-04-25T04:46:05.137 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.75922.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.75922.log.gz 2024-04-25T04:46:05.137 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104266.log 2024-04-25T04:46:05.138 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44732.log.gz 2024-04-25T04:46:05.138 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49335.log 2024-04-25T04:46:05.139 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104266.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104266.log.gz 2024-04-25T04:46:05.139 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.160746.log 2024-04-25T04:46:05.139 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49335.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49335.log.gz 2024-04-25T04:46:05.140 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95585.log 2024-04-25T04:46:05.140 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.160746.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.160746.log.gz 2024-04-25T04:46:05.141 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73291.log 2024-04-25T04:46:05.141 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95585.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95585.log.gz 2024-04-25T04:46:05.141 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47496.log 2024-04-25T04:46:05.142 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73291.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73291.log.gz 2024-04-25T04:46:05.142 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63526.log 2024-04-25T04:46:05.143 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47496.log.gz 2024-04-25T04:46:05.143 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133914.log 2024-04-25T04:46:05.144 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63526.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63526.log.gz 2024-04-25T04:46:05.144 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122488.log 2024-04-25T04:46:05.144 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133914.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133914.log.gz 2024-04-25T04:46:05.145 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132596.log 2024-04-25T04:46:05.145 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122488.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122488.log.gz 2024-04-25T04:46:05.146 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122090.log 2024-04-25T04:46:05.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132596.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132596.log.gz 2024-04-25T04:46:05.147 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60250.log 2024-04-25T04:46:05.147 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122090.log.gz 2024-04-25T04:46:05.148 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83570.log 2024-04-25T04:46:05.148 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60250.log.gz 2024-04-25T04:46:05.149 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62066.log 2024-04-25T04:46:05.149 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83570.log.gz 2024-04-25T04:46:05.149 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68656.log 2024-04-25T04:46:05.150 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62066.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62066.log.gz 2024-04-25T04:46:05.150 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44315.log 2024-04-25T04:46:05.151 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68656.log.gz 2024-04-25T04:46:05.151 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137542.log 2024-04-25T04:46:05.152 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44315.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44315.log.gz 2024-04-25T04:46:05.152 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95938.log 2024-04-25T04:46:05.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137542.log.gz 2024-04-25T04:46:05.153 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156194.log 2024-04-25T04:46:05.153 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95938.log.gz 2024-04-25T04:46:05.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66792.log 2024-04-25T04:46:05.154 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156194.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156194.log.gz 2024-04-25T04:46:05.155 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.53035.log 2024-04-25T04:46:05.155 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66792.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66792.log.gz 2024-04-25T04:46:05.156 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.157028.log 2024-04-25T04:46:05.157 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.53035.log: gzip -5 --verbose -- 0.0%/var/log/ceph/cluster1-client.admin.121329.log -- replaced with /var/log/ceph/cluster2-client.admin.53035.log.gz 2024-04-25T04:46:05.157 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-25T04:46:05.157 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.157028.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.157028.log.gz 2024-04-25T04:46:05.158 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71145.log 2024-04-25T04:46:05.158 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.121329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121329.log.gz 2024-04-25T04:46:05.159 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98652.log 2024-04-25T04:46:05.159 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71145.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71145.log.gz 2024-04-25T04:46:05.160 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61726.log 2024-04-25T04:46:05.161 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98652.log.gzgzip 2024-04-25T04:46:05.161 INFO:teuthology.orchestra.run.smithi188.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.156888.log 2024-04-25T04:46:05.162 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61726.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61726.log.gz 2024-04-25T04:46:05.162 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97199.log 2024-04-25T04:46:05.162 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.156888.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.156888.log.gz 2024-04-25T04:46:05.163 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87219.log 2024-04-25T04:46:05.163 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97199.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97199.log.gz 2024-04-25T04:46:05.164 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91768.log 2024-04-25T04:46:05.164 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87219.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87219.log.gz 2024-04-25T04:46:05.165 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124563.log 2024-04-25T04:46:05.165 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91768.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91768.log.gz 2024-04-25T04:46:05.165 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52383.log 2024-04-25T04:46:05.166 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.124563.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124563.log.gz 2024-04-25T04:46:05.166 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102171.log 2024-04-25T04:46:05.167 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52383.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52383.log.gz 2024-04-25T04:46:05.167 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84325.log 2024-04-25T04:46:05.168 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.102171.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102171.log.gz 2024-04-25T04:46:05.168 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161362.log 2024-04-25T04:46:05.169 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84325.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84325.log.gz 2024-04-25T04:46:05.169 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87738.log 2024-04-25T04:46:05.170 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103232.log 2024-04-25T04:46:05.170 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161362.log.gz 2024-04-25T04:46:05.171 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87738.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87738.log.gz 2024-04-25T04:46:05.171 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89106.log 2024-04-25T04:46:05.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103232.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103232.log.gz 2024-04-25T04:46:05.172 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.87641.log 2024-04-25T04:46:05.172 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89106.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89106.log.gz 2024-04-25T04:46:05.173 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42045.log 2024-04-25T04:46:05.174 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.87641.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.87641.log.gz 2024-04-25T04:46:05.174 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136994.log 2024-04-25T04:46:05.175 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.42045.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42045.log.gz 2024-04-25T04:46:05.175 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132491.log 2024-04-25T04:46:05.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.136994.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136994.log.gz 2024-04-25T04:46:05.176 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75235.log 2024-04-25T04:46:05.176 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132491.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132491.log.gz 2024-04-25T04:46:05.177 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59426.log 2024-04-25T04:46:05.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.75235.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75235.log.gz 2024-04-25T04:46:05.178 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116129.log 2024-04-25T04:46:05.178 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59426.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59426.log.gz 2024-04-25T04:46:05.178 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104433.log 2024-04-25T04:46:05.179 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116129.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116129.log.gz 2024-04-25T04:46:05.179 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160821.log 2024-04-25T04:46:05.180 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104433.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104433.log.gz 2024-04-25T04:46:05.180 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.89011.log 2024-04-25T04:46:05.181 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.160821.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160821.log.gz 2024-04-25T04:46:05.181 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.53479.log 2024-04-25T04:46:05.182 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.89011.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.89011.log.gz 2024-04-25T04:46:05.182 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94735.log 2024-04-25T04:46:05.183 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.53479.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.53479.log.gz 2024-04-25T04:46:05.183 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159396.log 2024-04-25T04:46:05.183 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94735.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94735.log.gz 2024-04-25T04:46:05.184 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.160385.log 2024-04-25T04:46:05.184 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159396.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159396.log.gz 2024-04-25T04:46:05.184 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85925.log 2024-04-25T04:46:05.185 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.160385.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.160385.log.gz 2024-04-25T04:46:05.185 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131289.log 2024-04-25T04:46:05.186 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85925.log.gz 2024-04-25T04:46:05.186 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47099.log 2024-04-25T04:46:05.187 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131289.log.gz 2024-04-25T04:46:05.187 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45413.log 2024-04-25T04:46:05.188 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47099.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47099.log.gz 2024-04-25T04:46:05.188 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104056.log 2024-04-25T04:46:05.188 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45413.log.gz 2024-04-25T04:46:05.189 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105262.log 2024-04-25T04:46:05.190 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104056.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104056.log.gz 2024-04-25T04:46:05.190 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44903.log 2024-04-25T04:46:05.191 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105262.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105262.log.gz 2024-04-25T04:46:05.191 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151265.log 2024-04-25T04:46:05.192 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44903.log.gz 2024-04-25T04:46:05.192 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77581.log 2024-04-25T04:46:05.193 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151265.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151265.log.gz 2024-04-25T04:46:05.193 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91208.log 2024-04-25T04:46:05.193 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77581.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77581.log.gz 2024-04-25T04:46:05.194 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62149.log 2024-04-25T04:46:05.194 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91208.log.gz 2024-04-25T04:46:05.195 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82009.log 2024-04-25T04:46:05.196 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92143.log 2024-04-25T04:46:05.196 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62149.log.gz 2024-04-25T04:46:05.197 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82009.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82009.log.gz 2024-04-25T04:46:05.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148719.log 2024-04-25T04:46:05.197 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92143.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92143.log.gz 2024-04-25T04:46:05.197 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69713.log 2024-04-25T04:46:05.198 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148719.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148719.log.gz 2024-04-25T04:46:05.198 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143745.log 2024-04-25T04:46:05.199 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69713.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69713.log.gz 2024-04-25T04:46:05.199 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141613.log 2024-04-25T04:46:05.200 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143745.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143745.log.gz 2024-04-25T04:46:05.200 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97012.log 2024-04-25T04:46:05.201 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141613.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141613.log.gz 2024-04-25T04:46:05.201 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44628.log 2024-04-25T04:46:05.201 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97012.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97012.log.gz 2024-04-25T04:46:05.202 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103120.log 2024-04-25T04:46:05.202 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44628.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44628.log.gz 2024-04-25T04:46:05.203 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115158.log 2024-04-25T04:46:05.204 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.103120.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103120.log.gz 2024-04-25T04:46:05.204 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128336.log 2024-04-25T04:46:05.205 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115158.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115158.log.gz 2024-04-25T04:46:05.205 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44839.log 2024-04-25T04:46:05.205 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128336.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128336.log.gz 2024-04-25T04:46:05.206 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115940.log 2024-04-25T04:46:05.206 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44839.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44839.log.gz 2024-04-25T04:46:05.207 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81610.log 2024-04-25T04:46:05.207 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115940.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115940.log.gz 2024-04-25T04:46:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124986.log 2024-04-25T04:46:05.208 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81610.log.gz 2024-04-25T04:46:05.209 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142691.log 2024-04-25T04:46:05.209 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.124986.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124986.log.gz 2024-04-25T04:46:05.209 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150720.log 2024-04-25T04:46:05.210 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142691.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142691.log.gz 2024-04-25T04:46:05.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79630.log 2024-04-25T04:46:05.211 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150720.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150720.log.gz 2024-04-25T04:46:05.211 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67305.log 2024-04-25T04:46:05.212 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79630.log.gz 2024-04-25T04:46:05.212 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111700.log 2024-04-25T04:46:05.213 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67305.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67305.log.gz 2024-04-25T04:46:05.213 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137504.log 2024-04-25T04:46:05.214 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111700.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111700.log.gz 2024-04-25T04:46:05.214 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80322.log 2024-04-25T04:46:05.215 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.137504.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137504.log.gz 2024-04-25T04:46:05.215 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130302.log 2024-04-25T04:46:05.215 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80322.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80322.log.gz 2024-04-25T04:46:05.216 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79841.log 2024-04-25T04:46:05.217 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130302.log.gz 2024-04-25T04:46:05.217 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115606.log 2024-04-25T04:46:05.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.79841.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79841.log.gz 2024-04-25T04:46:05.218 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158295.log 2024-04-25T04:46:05.219 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115606.log.gz 2024-04-25T04:46:05.219 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46027.log 2024-04-25T04:46:05.220 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158295.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158295.log.gz 2024-04-25T04:46:05.220 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-04-25T04:46:05.221 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46027.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46027.log.gz 2024-04-25T04:46:05.221 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90943.log 2024-04-25T04:46:05.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.100240.log 2024-04-25T04:46:05.222 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90943.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90943.log.gz 2024-04-25T04:46:05.223 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155880.log 2024-04-25T04:46:05.223 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.100240.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.100240.log.gz 2024-04-25T04:46:05.224 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58713.log 2024-04-25T04:46:05.224 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.155880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.155880.log.gz 2024-04-25T04:46:05.224 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120260.log 2024-04-25T04:46:05.225 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58713.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58713.log.gz 2024-04-25T04:46:05.225 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99617.log 2024-04-25T04:46:05.226 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120260.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120260.log.gz 2024-04-25T04:46:05.226 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146367.log 2024-04-25T04:46:05.227 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99617.log.gz 2024-04-25T04:46:05.227 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43434.log 2024-04-25T04:46:05.228 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146367.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146367.log.gz 2024-04-25T04:46:05.228 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66013.log 2024-04-25T04:46:05.229 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43434.log.gz 2024-04-25T04:46:05.229 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115542.log 2024-04-25T04:46:05.229 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.66013.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66013.log.gz 2024-04-25T04:46:05.230 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62091.log 2024-04-25T04:46:05.230 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115542.log.gz 2024-04-25T04:46:05.231 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44318.log 2024-04-25T04:46:05.232 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62091.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62091.log.gz 2024-04-25T04:46:05.232 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125369.log 2024-04-25T04:46:05.232 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44318.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44318.log.gz 2024-04-25T04:46:05.233 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53990.log 2024-04-25T04:46:05.233 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.125369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125369.log.gz 2024-04-25T04:46:05.233 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117144.log 2024-04-25T04:46:05.234 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53990.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53990.log.gz 2024-04-25T04:46:05.234 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48869.log 2024-04-25T04:46:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117144.log.gz 2024-04-25T04:46:05.235 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58040.log 2024-04-25T04:46:05.236 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48869.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48869.log.gz 2024-04-25T04:46:05.236 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93392.log 2024-04-25T04:46:05.237 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58040.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58040.log.gz 2024-04-25T04:46:05.237 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113766.log 2024-04-25T04:46:05.238 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93392.log.gz 2024-04-25T04:46:05.238 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148007.log 2024-04-25T04:46:05.238 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113766.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113766.log.gz 2024-04-25T04:46:05.239 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-04-25T04:46:05.239 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.148007.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.148007.log.gz 2024-04-25T04:46:05.240 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154391.log 2024-04-25T04:46:05.240 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85830.log 2024-04-25T04:46:05.241 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154391.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154391.log.gz 2024-04-25T04:46:05.241 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87259.log 2024-04-25T04:46:05.242 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.85830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.85830.log.gz 2024-04-25T04:46:05.242 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153623.log 2024-04-25T04:46:05.243 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87259.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87259.log.gz 2024-04-25T04:46:05.243 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130300.log 2024-04-25T04:46:05.243 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153623.log.gz 2024-04-25T04:46:05.244 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147415.log 2024-04-25T04:46:05.244 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130300.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130300.log.gz 2024-04-25T04:46:05.245 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152488.log 2024-04-25T04:46:05.245 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.147415.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.147415.log.gz 2024-04-25T04:46:05.246 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62071.log 2024-04-25T04:46:05.247 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152488.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152488.log.gz 2024-04-25T04:46:05.247 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94420.log 2024-04-25T04:46:05.247 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.62071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62071.log.gz 2024-04-25T04:46:05.248 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83126.log 2024-04-25T04:46:05.248 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94420.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94420.log.gz 2024-04-25T04:46:05.248 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118090.log 2024-04-25T04:46:05.249 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.83126.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83126.log.gz 2024-04-25T04:46:05.249 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143454.log 2024-04-25T04:46:05.250 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118090.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118090.log.gz 2024-04-25T04:46:05.250 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132193.log 2024-04-25T04:46:05.251 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143454.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143454.log.gz 2024-04-25T04:46:05.251 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95436.log 2024-04-25T04:46:05.252 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132193.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132193.log.gz 2024-04-25T04:46:05.252 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156570.log 2024-04-25T04:46:05.253 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95436.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95436.log.gz 2024-04-25T04:46:05.253 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112945.log 2024-04-25T04:46:05.254 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156570.log.gz 2024-04-25T04:46:05.254 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59977.log 2024-04-25T04:46:05.255 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.112945.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112945.log.gz 2024-04-25T04:46:05.255 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153124.log 2024-04-25T04:46:05.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.59977.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59977.log.gz 2024-04-25T04:46:05.256 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80173.log 2024-04-25T04:46:05.256 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153124.log.gz 2024-04-25T04:46:05.257 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44800.log 2024-04-25T04:46:05.257 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.80173.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80173.log.gz 2024-04-25T04:46:05.258 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106604.log 2024-04-25T04:46:05.258 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.44800.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44800.log.gz 2024-04-25T04:46:05.259 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49502.log 2024-04-25T04:46:05.260 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47902.log 2024-04-25T04:46:05.260 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106604.log.gz 2024-04-25T04:46:05.260 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.49502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49502.log.gz 2024-04-25T04:46:05.260 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159544.log 2024-04-25T04:46:05.261 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47902.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47902.log.gz 2024-04-25T04:46:05.261 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132549.log 2024-04-25T04:46:05.262 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159544.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159544.log.gz 2024-04-25T04:46:05.262 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134221.log 2024-04-25T04:46:05.263 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132549.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132549.log.gz 2024-04-25T04:46:05.263 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95711.log 2024-04-25T04:46:05.264 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.134221.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.134221.log.gz 2024-04-25T04:46:05.264 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159584.log 2024-04-25T04:46:05.265 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.95711.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.95711.log.gz 2024-04-25T04:46:05.265 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122332.log 2024-04-25T04:46:05.266 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.159584.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159584.log.gz 2024-04-25T04:46:05.266 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42746.log 2024-04-25T04:46:05.267 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122332.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122332.log.gz 2024-04-25T04:46:05.267 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106710.log 2024-04-25T04:46:05.268 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42746.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42746.log.gz 2024-04-25T04:46:05.268 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42215.log 2024-04-25T04:46:05.269 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106710.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106710.log.gz 2024-04-25T04:46:05.269 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119100.log 2024-04-25T04:46:05.270 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42215.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42215.log.gz 2024-04-25T04:46:05.270 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99583.log 2024-04-25T04:46:05.270 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.119100.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119100.log.gz 2024-04-25T04:46:05.271 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146536.log 2024-04-25T04:46:05.271 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.99583.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99583.log.gz 2024-04-25T04:46:05.272 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142862.log 2024-04-25T04:46:05.272 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.146536.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146536.log.gz 2024-04-25T04:46:05.272 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138866.log 2024-04-25T04:46:05.273 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142862.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142862.log.gz 2024-04-25T04:46:05.273 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152509.log 2024-04-25T04:46:05.274 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138866.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138866.log.gz 2024-04-25T04:46:05.274 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132659.log 2024-04-25T04:46:05.275 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152509.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152509.log.gz 2024-04-25T04:46:05.275 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47903.log 2024-04-25T04:46:05.276 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.132659.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132659.log.gz 2024-04-25T04:46:05.276 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73630.log 2024-04-25T04:46:05.276 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.47903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47903.log.gz 2024-04-25T04:46:05.277 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133869.log 2024-04-25T04:46:05.277 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73630.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73630.log.gz 2024-04-25T04:46:05.278 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145062.log 2024-04-25T04:46:05.278 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.133869.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133869.log.gz 2024-04-25T04:46:05.279 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144332.log 2024-04-25T04:46:05.279 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145062.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145062.log.gz 2024-04-25T04:46:05.280 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154850.log 2024-04-25T04:46:05.280 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144332.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144332.log.gz 2024-04-25T04:46:05.281 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79289.log 2024-04-25T04:46:05.281 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154850.log.gz 2024-04-25T04:46:05.282 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159921.log 2024-04-25T04:46:05.282 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.79289.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79289.log.gz 2024-04-25T04:46:05.283 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96844.log 2024-04-25T04:46:05.283 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.159921.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.159921.log.gz 2024-04-25T04:46:05.283 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113962.log 2024-04-25T04:46:05.284 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96844.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96844.log.gz 2024-04-25T04:46:05.284 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122044.log 2024-04-25T04:46:05.285 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113962.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113962.log.gz 2024-04-25T04:46:05.285 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108906.log 2024-04-25T04:46:05.286 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.122044.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122044.log.gz 2024-04-25T04:46:05.286 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128783.log 2024-04-25T04:46:05.287 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108906.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108906.log.gz 2024-04-25T04:46:05.287 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40710.log 2024-04-25T04:46:05.288 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128783.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128783.log.gz 2024-04-25T04:46:05.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112081.log 2024-04-25T04:46:05.289 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.40710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40710.log.gz 2024-04-25T04:46:05.289 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48208.log 2024-04-25T04:46:05.290 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112081.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112081.log.gz 2024-04-25T04:46:05.290 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.156596.log 2024-04-25T04:46:05.291 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48208.log.gz 2024-04-25T04:46:05.291 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81361.log 2024-04-25T04:46:05.291 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.156596.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.156596.log.gz 2024-04-25T04:46:05.292 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92952.log 2024-04-25T04:46:05.292 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81361.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81361.log.gz 2024-04-25T04:46:05.292 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61070.log 2024-04-25T04:46:05.293 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92952.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92952.log.gz 2024-04-25T04:46:05.293 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143789.log 2024-04-25T04:46:05.294 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.61070.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61070.log.gz 2024-04-25T04:46:05.294 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96426.log 2024-04-25T04:46:05.295 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143789.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143789.log.gz 2024-04-25T04:46:05.295 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133103.log 2024-04-25T04:46:05.296 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96426.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96426.log.gz 2024-04-25T04:46:05.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45963.log 2024-04-25T04:46:05.297 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133103.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133103.log.gz 2024-04-25T04:46:05.297 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46067.log 2024-04-25T04:46:05.298 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45963.log.gz 2024-04-25T04:46:05.298 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152071.log 2024-04-25T04:46:05.298 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.46067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46067.log.gz 2024-04-25T04:46:05.299 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62327.log 2024-04-25T04:46:05.299 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152071.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152071.log.gz 2024-04-25T04:46:05.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.147753.log 2024-04-25T04:46:05.300 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.62327.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62327.log.gz 2024-04-25T04:46:05.300 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121000.log 2024-04-25T04:46:05.301 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.147753.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.147753.log.gz 2024-04-25T04:46:05.301 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158374.log 2024-04-25T04:46:05.302 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.121000.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121000.log.gz 2024-04-25T04:46:05.302 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68654.log 2024-04-25T04:46:05.303 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.158374.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158374.log.gz 2024-04-25T04:46:05.303 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120285.log 2024-04-25T04:46:05.304 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.68654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68654.log.gz 2024-04-25T04:46:05.304 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74868.log 2024-04-25T04:46:05.305 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120285.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120285.log.gz 2024-04-25T04:46:05.305 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130465.log 2024-04-25T04:46:05.305 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.74868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74868.log.gz 2024-04-25T04:46:05.306 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130381.log 2024-04-25T04:46:05.306 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130465.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130465.log.gz 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152347.log 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89749.log 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130381.log.gz 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.152347.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152347.log.gz 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57829.log 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89749.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89749.log.gz 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161117.log 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.57829.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57829.log.gz 2024-04-25T04:46:05.311 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109604.log 2024-04-25T04:46:05.312 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.161117.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161117.log.gz 2024-04-25T04:46:05.312 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69483.log 2024-04-25T04:46:05.313 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.109604.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109604.log.gz 2024-04-25T04:46:05.313 INFO:teuthology.orchestra.run.smithi188.stderr: 92.5% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-04-25T04:46:05.313 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.124286.log 2024-04-25T04:46:05.313 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.69483.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69483.log.gz 2024-04-25T04:46:05.313 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82514.log 2024-04-25T04:46:05.314 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.124286.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.124286.log.gz 2024-04-25T04:46:05.314 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96970.log 2024-04-25T04:46:05.315 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.82514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82514.log.gz 2024-04-25T04:46:05.315 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86710.log 2024-04-25T04:46:05.316 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96970.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96970.log.gz 2024-04-25T04:46:05.316 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76874.log 2024-04-25T04:46:05.317 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86710.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86710.log.gz 2024-04-25T04:46:05.317 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97116.log 2024-04-25T04:46:05.318 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76874.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76874.log.gz 2024-04-25T04:46:05.318 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48006.log 2024-04-25T04:46:05.318 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.97116.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97116.log.gz 2024-04-25T04:46:05.319 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.89387.log 2024-04-25T04:46:05.319 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.48006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48006.log.gz 2024-04-25T04:46:05.320 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58988.log 2024-04-25T04:46:05.320 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.89387.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.89387.log.gz 2024-04-25T04:46:05.320 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84138.log 2024-04-25T04:46:05.321 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58988.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58988.log.gz 2024-04-25T04:46:05.321 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114230.log 2024-04-25T04:46:05.322 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84138.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84138.log.gz 2024-04-25T04:46:05.322 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110638.log 2024-04-25T04:46:05.323 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.114230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114230.log.gz 2024-04-25T04:46:05.323 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101369.log 2024-04-25T04:46:05.324 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.110638.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110638.log.gz 2024-04-25T04:46:05.324 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96911.log 2024-04-25T04:46:05.325 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.101369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101369.log.gz 2024-04-25T04:46:05.325 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108355.log 2024-04-25T04:46:05.326 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96911.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96911.log.gz 2024-04-25T04:46:05.326 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.137186.log 2024-04-25T04:46:05.327 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.108355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108355.log.gz 2024-04-25T04:46:05.327 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150615.log 2024-04-25T04:46:05.328 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.137186.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.137186.log.gz 2024-04-25T04:46:05.328 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93244.log 2024-04-25T04:46:05.329 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.150615.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150615.log.gz 2024-04-25T04:46:05.329 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153375.log 2024-04-25T04:46:05.330 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93244.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93244.log.gz 2024-04-25T04:46:05.330 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81231.log 2024-04-25T04:46:05.331 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153375.log.gz 2024-04-25T04:46:05.331 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53762.log 2024-04-25T04:46:05.331 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81231.log.gz 2024-04-25T04:46:05.332 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139411.log 2024-04-25T04:46:05.332 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.53762.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53762.log.gz 2024-04-25T04:46:05.333 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52467.log 2024-04-25T04:46:05.333 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.139411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139411.log.gz 2024-04-25T04:46:05.334 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138581.log 2024-04-25T04:46:05.334 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52467.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52467.log.gz 2024-04-25T04:46:05.335 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86931.log 2024-04-25T04:46:05.335 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.138581.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138581.log.gz 2024-04-25T04:46:05.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81880.log 2024-04-25T04:46:05.336 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86931.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86931.log.gz 2024-04-25T04:46:05.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73734.log 2024-04-25T04:46:05.337 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.81880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81880.log.gz 2024-04-25T04:46:05.337 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129813.log 2024-04-25T04:46:05.338 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.73734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73734.log.gz 2024-04-25T04:46:05.338 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98909.log 2024-04-25T04:46:05.339 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129813.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129813.log.gz 2024-04-25T04:46:05.339 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60671.log 2024-04-25T04:46:05.340 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.98909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98909.log.gz 2024-04-25T04:46:05.340 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129078.log 2024-04-25T04:46:05.341 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.60671.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60671.log.gz 2024-04-25T04:46:05.341 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125462.log 2024-04-25T04:46:05.341 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129078.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129078.log.gz 2024-04-25T04:46:05.342 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.158926.log 2024-04-25T04:46:05.343 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.125462.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125462.log.gz 2024-04-25T04:46:05.343 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153330.log 2024-04-25T04:46:05.344 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.158926.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.158926.log.gz 2024-04-25T04:46:05.344 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.102059.log 2024-04-25T04:46:05.345 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.153330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153330.log.gz 2024-04-25T04:46:05.345 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104142.log 2024-04-25T04:46:05.346 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.102059.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.102059.log.gz 2024-04-25T04:46:05.346 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149718.log 2024-04-25T04:46:05.347 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.104142.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104142.log.gz 2024-04-25T04:46:05.347 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116444.log 2024-04-25T04:46:05.347 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.149718.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149718.log.gz 2024-04-25T04:46:05.348 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144415.log 2024-04-25T04:46:05.348 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.116444.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116444.log.gz 2024-04-25T04:46:05.349 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91875.log 2024-04-25T04:46:05.349 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144415.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144415.log.gz 2024-04-25T04:46:05.350 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115290.log 2024-04-25T04:46:05.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.91875.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91875.log.gz 2024-04-25T04:46:05.351 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118993.log 2024-04-25T04:46:05.351 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.115290.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115290.log.gz 2024-04-25T04:46:05.352 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58880.log 2024-04-25T04:46:05.352 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118993.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118993.log.gz 2024-04-25T04:46:05.353 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151851.log 2024-04-25T04:46:05.353 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58880.log.gz 2024-04-25T04:46:05.354 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105830.log 2024-04-25T04:46:05.354 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.151851.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.151851.log.gz 2024-04-25T04:46:05.354 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.135910.log 2024-04-25T04:46:05.355 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.105830.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105830.log.gz 2024-04-25T04:46:05.356 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31546.log 2024-04-25T04:46:05.356 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.135910.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.135910.log.gz 2024-04-25T04:46:05.356 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76314.log 2024-04-25T04:46:05.357 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/ceph.tmp-client.admin.31546.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31546.log.gz 2024-04-25T04:46:05.357 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84622.log 2024-04-25T04:46:05.358 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76314.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76314.log.gz 2024-04-25T04:46:05.358 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129016.log 2024-04-25T04:46:05.359 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.84622.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84622.log.gz 2024-04-25T04:46:05.359 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70425.log 2024-04-25T04:46:05.360 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.129016.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.129016.log.gz 2024-04-25T04:46:05.360 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92334.log 2024-04-25T04:46:05.360 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70425.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70425.log.gz 2024-04-25T04:46:05.361 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106818.log 2024-04-25T04:46:05.361 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.92334.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92334.log.gz 2024-04-25T04:46:05.362 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58836.log 2024-04-25T04:46:05.362 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106818.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106818.log.gz 2024-04-25T04:46:05.363 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113067.log 2024-04-25T04:46:05.363 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.58836.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58836.log.gz 2024-04-25T04:46:05.364 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128908.log 2024-04-25T04:46:05.364 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113067.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113067.log.gz 2024-04-25T04:46:05.365 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.146018.log 2024-04-25T04:46:05.366 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128908.log.gz 2024-04-25T04:46:05.366 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120833.log 2024-04-25T04:46:05.366 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.146018.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.146018.log.gz 2024-04-25T04:46:05.367 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96070.log 2024-04-25T04:46:05.367 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.120833.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120833.log.gz 2024-04-25T04:46:05.368 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154726.log 2024-04-25T04:46:05.368 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96070.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96070.log.gz 2024-04-25T04:46:05.368 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94252.log 2024-04-25T04:46:05.369 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.154726.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154726.log.gz 2024-04-25T04:46:05.369 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71882.log 2024-04-25T04:46:05.370 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.94252.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94252.log.gz 2024-04-25T04:46:05.370 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71621.log 2024-04-25T04:46:05.371 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71882.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71882.log.gz 2024-04-25T04:46:05.371 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131187.log 2024-04-25T04:46:05.372 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.71621.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71621.log.gz 2024-04-25T04:46:05.372 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63311.log 2024-04-25T04:46:05.373 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131187.log.gz 2024-04-25T04:46:05.373 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45966.log 2024-04-25T04:46:05.374 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.63311.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63311.log.gz 2024-04-25T04:46:05.374 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86467.log 2024-04-25T04:46:05.374 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.45966.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45966.log.gz 2024-04-25T04:46:05.375 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67187.log 2024-04-25T04:46:05.375 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86467.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86467.log.gz 2024-04-25T04:46:05.375 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96910.log 2024-04-25T04:46:05.376 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.67187.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67187.log.gz 2024-04-25T04:46:05.376 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43955.log 2024-04-25T04:46:05.377 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.96910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96910.log.gz 2024-04-25T04:46:05.377 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90414.log 2024-04-25T04:46:05.378 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43955.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43955.log.gz 2024-04-25T04:46:05.378 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.88426.log 2024-04-25T04:46:05.379 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90414.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90414.log.gz 2024-04-25T04:46:05.379 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.65557.log 2024-04-25T04:46:05.380 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.88426.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.88426.log.gz 2024-04-25T04:46:05.380 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70847.log 2024-04-25T04:46:05.381 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.65557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128167.log 2024-04-25T04:46:05.381 INFO:teuthology.orchestra.run.smithi188.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.65557.log.gz 2024-04-25T04:46:05.382 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70847.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70847.log.gz 2024-04-25T04:46:05.382 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93665.log 2024-04-25T04:46:05.382 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128167.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128167.log.gz 2024-04-25T04:46:05.383 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128724.log 2024-04-25T04:46:05.383 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.93665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93665.log.gz 2024-04-25T04:46:05.384 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131815.log 2024-04-25T04:46:05.385 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128724.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128724.log.gz 2024-04-25T04:46:05.385 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142077.log 2024-04-25T04:46:05.385 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.131815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131815.log.gz 2024-04-25T04:46:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143957.log 2024-04-25T04:46:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.142077.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142077.log.gz 2024-04-25T04:46:05.386 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106603.log 2024-04-25T04:46:05.387 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.143957.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143957.log.gz 2024-04-25T04:46:05.387 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118196.log 2024-04-25T04:46:05.388 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.106603.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.106603.log.gz 2024-04-25T04:46:05.388 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118072.log 2024-04-25T04:46:05.389 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118196.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118196.log.gz 2024-04-25T04:46:05.389 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107622.log 2024-04-25T04:46:05.390 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.118072.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118072.log.gz 2024-04-25T04:46:05.390 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144913.log 2024-04-25T04:46:05.391 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.107622.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107622.log.gz 2024-04-25T04:46:05.391 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86821.log 2024-04-25T04:46:05.392 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.144913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144913.log.gz 2024-04-25T04:46:05.392 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70049.log 2024-04-25T04:46:05.393 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.86821.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86821.log.gz 2024-04-25T04:46:05.393 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.77923.log 2024-04-25T04:46:05.394 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.70049.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70049.log.gz 2024-04-25T04:46:05.394 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130628.log 2024-04-25T04:46:05.395 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.77923.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.77923.log.gz 2024-04-25T04:46:05.395 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.76311.log 2024-04-25T04:46:05.396 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.130628.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130628.log.gz 2024-04-25T04:46:05.396 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.111051.log 2024-04-25T04:46:05.397 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.76311.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.76311.log.gz 2024-04-25T04:46:05.397 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41668.log 2024-04-25T04:46:05.397 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.111051.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.111051.log.gz 2024-04-25T04:46:05.398 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42938.log 2024-04-25T04:46:05.399 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.41668.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.41668.log.gz 2024-04-25T04:46:05.399 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.147703.log 2024-04-25T04:46:05.399 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.42938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42938.log.gz 2024-04-25T04:46:05.400 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145341.log 2024-04-25T04:46:05.401 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.147703.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.147703.log.gz 2024-04-25T04:46:05.401 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133362.log 2024-04-25T04:46:05.402 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.145341.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.145341.log.gz 2024-04-25T04:46:05.402 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78073.log 2024-04-25T04:46:05.403 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.133362.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133362.log.gz 2024-04-25T04:46:05.403 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141363.log 2024-04-25T04:46:05.404 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.78073.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78073.log.gz 2024-04-25T04:46:05.404 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43057.log 2024-04-25T04:46:05.405 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.141363.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141363.log.gz 2024-04-25T04:46:05.405 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90496.log 2024-04-25T04:46:05.405 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.43057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43057.log.gz 2024-04-25T04:46:05.406 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138161.log 2024-04-25T04:46:05.406 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.90496.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90496.log.gz 2024-04-25T04:46:05.407 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148854.log 2024-04-25T04:46:05.407 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.138161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138161.log.gz 2024-04-25T04:46:05.407 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87306.log 2024-04-25T04:46:05.408 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.148854.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148854.log.gz 2024-04-25T04:46:05.408 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.112306.log 2024-04-25T04:46:05.409 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.admin.87306.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87306.log.gz 2024-04-25T04:46:05.409 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55461.log 2024-04-25T04:46:05.410 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.112306.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.112306.log.gz 2024-04-25T04:46:05.410 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117903.log 2024-04-25T04:46:05.411 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55461.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55461.log.gz 2024-04-25T04:46:05.411 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55526.log 2024-04-25T04:46:05.412 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.117903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117903.log.gz 2024-04-25T04:46:05.412 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.52000.log 2024-04-25T04:46:05.413 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.55526.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55526.log.gz 2024-04-25T04:46:05.413 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.63702.log 2024-04-25T04:46:05.414 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.52000.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.52000.log.gz 2024-04-25T04:46:05.414 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113803.log 2024-04-25T04:46:05.415 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.admin.63702.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.63702.log.gz 2024-04-25T04:46:05.415 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.88108.log 2024-04-25T04:46:05.416 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.113803.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113803.log.gz 2024-04-25T04:46:05.416 INFO:teuthology.orchestra.run.smithi188.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128315.log 2024-04-25T04:46:05.416 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster2-client.mirror.88108.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.88108.log.gz 2024-04-25T04:46:05.417 INFO:teuthology.orchestra.run.smithi188.stderr:/var/log/ceph/cluster1-client.mirror.128315.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128315.log.gz 2024-04-25T04:46:17.667 INFO:teuthology.orchestra.run.smithi173.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-04-25T04:46:36.226 INFO:teuthology.orchestra.run.smithi188.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-04-25T04:51:10.568 INFO:teuthology.orchestra.run.smithi188.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-04-25T04:51:26.083 INFO:teuthology.orchestra.run.smithi188.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-04-25T04:51:59.497 INFO:teuthology.orchestra.run.smithi188.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-04-25T04:51:59.501 INFO:teuthology.orchestra.run.smithi188.stderr: 2024-04-25T04:51:59.501 INFO:teuthology.orchestra.run.smithi188.stderr:real 5m58.813s 2024-04-25T04:51:59.501 INFO:teuthology.orchestra.run.smithi188.stderr:user 2m12.313s 2024-04-25T04:51:59.501 INFO:teuthology.orchestra.run.smithi188.stderr:sys 0m32.857s 2024-04-25T04:58:57.029 INFO:teuthology.orchestra.run.smithi173.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-04-25T04:59:39.635 INFO:teuthology.orchestra.run.smithi173.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-04-25T05:00:31.437 INFO:teuthology.orchestra.run.smithi173.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-04-25T05:00:31.444 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-25T05:00:31.444 INFO:teuthology.orchestra.run.smithi173.stderr:real 14m30.761s 2024-04-25T05:00:31.444 INFO:teuthology.orchestra.run.smithi173.stderr:user 4m52.184s 2024-04-25T05:00:31.444 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m52.121s 2024-04-25T05:00:31.445 INFO:tasks.ceph:Archiving logs... 2024-04-25T05:00:31.445 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/remote/smithi173/log 2024-04-25T05:00:31.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T05:01:12.373 DEBUG:teuthology.misc:Transferring archived files from smithi188:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/remote/smithi188/log 2024-04-25T05:01:12.375 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T05:01:34.762 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T05:01:34.773 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T05:01:34.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T05:01:34.779 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T05:01:34.937 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T05:01:34.946 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T05:01:34.946 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T05:01:34.950 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T05:01:34.991 INFO:teuthology.orchestra.run.smithi173.stderr:bash: ntpq: command not found 2024-04-25T05:01:34.994 INFO:teuthology.orchestra.run.smithi188.stderr:bash: ntpq: command not found 2024-04-25T05:01:35.064 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T05:01:35.064 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-25T05:01:35.064 INFO:teuthology.orchestra.run.smithi173.stdout:^- hv01.front.sepia.ceph.com 2 7 377 115 +703us[ +747us] +/- 579ms 2024-04-25T05:01:35.064 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 7 377 52 -746us[ -702us] +/- 55ms 2024-04-25T05:01:35.064 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 181 +924us[ +969us] +/- 73ms 2024-04-25T05:01:35.064 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:01:35.065 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T05:01:35.065 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-04-25T05:01:35.065 INFO:teuthology.orchestra.run.smithi188.stdout:^- hv01.front.sepia.ceph.com 2 6 377 55 +723us[ +765us] +/- 580ms 2024-04-25T05:01:35.065 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -759us[ -717us] +/- 56ms 2024-04-25T05:01:35.065 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +936us[ +936us] +/- 75ms 2024-04-25T05:01:35.066 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T05:01:35.066 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T05:01:35.078 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T05:01:35.078 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T05:01:35.122 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T05:01:35.289 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T05:01:35.508 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-25T05:01:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:01:35.579 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-25T05:01:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:01:35.651 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T05:01:35.661 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T05:01:35.703 INFO:teuthology.task.internal:Duration was 6633.266273 seconds 2024-04-25T05:01:35.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T05:01:35.737 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T05:01:35.738 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T05:01:35.742 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T05:01:35.881 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T05:01:35.894 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T05:01:36.170 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T05:01:36.170 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T05:01:36.171 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T05:01:36.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-04-25T05:01:36.234 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T05:01:36.332 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T05:01:36.332 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T05:01:36.338 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T05:01:36.465 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T05:01:36.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T05:01:36.470 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T05:01:36.618 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T05:01:36.632 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T05:01:36.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T05:01:36.702 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T05:01:36.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T05:01:36.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T05:01:36.809 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T05:01:36.889 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-04-25T05:01:36.891 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-04-25T05:01:36.929 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T05:01:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:01:36.972 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T05:01:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T05:01:37.015 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T05:01:37.025 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T05:01:37.026 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/remote/smithi173 2024-04-25T05:01:37.026 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T05:01:37.104 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:48:03-krbd-squid-testing-default-smithi/7672443/remote/smithi188 2024-04-25T05:01:37.105 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T05:01:37.185 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T05:01:37.185 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T05:01:37.189 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T05:01:37.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T05:01:37.246 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T05:01:37.246 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T05:01:37.303 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T05:01:37.304 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T05:01:37.308 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T05:01:37.348 INFO:teuthology.orchestra.run.smithi173.stdout: 524317 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 05:01 /home/ubuntu/cephtest 2024-04-25T05:01:37.352 INFO:teuthology.orchestra.run.smithi188.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 05:01 /home/ubuntu/cephtest 2024-04-25T05:01:37.355 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T05:01:37.370 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T05:01:37.453 INFO:teuthology.run:Summary data: cluster1.mon.a-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 cluster2.mon.a-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: krbd/mirror/{bluestore-bitmap clusters/2-node conf install/ceph ms_mode$/{crc-rxbounce} tasks/compare-mirror-images} duration: 6633.266273021698 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T05:01:37.453 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T05:01:37.537 INFO:teuthology.run:pass